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ABSTRAC!jr . . / 

P.resiBnt6d in this report selected findings of the 
Income! Verification Pilot Project (IVPP), an investigation examining 
misrenorting of applicant income and family size on applications for 
government-spon,sored school meal benefits* Ais reported here ji! Phase II 
of the pjynect provided for a comprehensive assessment of specific 
quality assurance procedures in a national sample of school *districts 
during the 1982-83 school year. Participating were 16 states and a • 
sample of 114 school districts: 29. experimental sites where seven 
.quality assurance procedures were test^ed and 85 control sites. After 
the first chapter' s^brief introduction/ chapter 2 explains the / 
various quality assurance procedures examined. Chapter^ 3 and 4 
describe the research design and data collection proce^dures. Chapter 
5 presents findings concerning the feasibility and effects of free, 
andvireduced-price school meal applications that were /developed 
specifically for use in ^he IVPP. Findings on feasibility and effects 
oL requiring income documentation wit)i the free and reduced-price 
school meal applications are reported in chapter 6 j Chapter 7 
presents findings on the feasibility and effects ot quality assurance 
procedures used after sdhool meal applications are proce^ssed. 
Finally, results and conclusions of the pilot pr^oject are summarized 
in chapter 8, Appendices include the IVPP application for free and 
reduced-price school meals, a summary of experimental school 
districts' characteristics, a measurement of quality assurance 
procedure costs, and a nonresponse analysis. (RH) 
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' 'EXECUjiVE SUMMARY 
. Pilot Study Background , ' - ' y 



This summary presents concluslons frdm a Congre^^^ 
of procedjures designed to reduce fraud and abuse* in the fedeirally-suppprted school 
meal prdgrams. It focuses on the results of, a national test of seven quality 
? \a38urancp procedures designed* 'to prevent and detect misreportirig on free and 
'Reduced-price school meal applications. ij ; * ' , v ^ > . 

Prior studies, 'including one conducted in 1981 by the U-S. Department of 
Agriculture's (USDA)* Office of the Inspector General, / have estimated that ' 
,appr|pximatel>!^'one of every four recipients of free and reduced-^rice School meals 
was' receiving benefits he was not -dntitled to receive, DurjKg the 1,981-82 sqhool 
yfear, in a srpaU sample of school districts, the Department tested simple methods to 
jdQter applicant misreporting and reduce the error rate, "Uiese 'methods included a 
slight modification to the free and redgced-price meal application then In use, 
providing information about income that should be reported on the application, and 
issuing a ^warning that the .applications might be verified, /Although the/ sample of 
school districts was not nationally representative, these simple procedures reduced 
the errorrate to 17.4 percent, ' 

During the 1982-83/ sch^^ the~ Department implemented d large- scale, / 

^jfiationally representative- test or seven quality assurance procedures. These quality. 
\ €i9surance procedures were ci^^^ (1) to reduce the error r^te by de^errirtg * 
household misreporting on the application, and (2) to enable school districts to verify 
information oh the applicatlo"^ and, thereby, ensure that program pa^ticiparlts . ; 
appropriately receive the 'meals; to which they were, entitlgd. The major oljjectlves 
: of this targe-scale test, known ^s Phase II of the Income Verification Pilot Project : 7 
^ (IVPP), were to: 



determine the jpapability of tha^ quality assurance procedures to reduce 
erroi;^rates by'^eterring and detecting applicant misreporting; 

determine if it was feasible for school districts to conduct the quality ; 
assurance procedures; 

measure the costs to perform and savihgs that result from each quality ' 

assur.ance procedure; and ,^ V V , , ' 
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• ■ examine the barrier effaota, If any, of . t^^^ prooeciurea on program 
' partlalpantr and their famllleai Barrlera are the obataeleg poaed by the 
• proceduraa 'that result In eligible houaehQldP not receiving benefltfl. 

Quality AoDurance Prooedurea ,1/ \* \ 

The aevenjquallty aaauranoe procedupea Implemented In.thla pilot atudy werei 

,1, . A Revlaed Application Form dealgn^d to dker mlarepprtlng and to provide 
data needed to' ouppott the legal 'and rnfOrmatlphal rqq'ulremento of 
aubaequent verification procedurea, (The application la referred^o aa. the 
IVPP application.) i ' l > ' ^ ■ ' < 

bocumentatloh the IVPP 'ApDHcatlon to support Information about 
earnbd jnccme' Or othar benefit ,program\ eligibility ' (e.g,, food atamps) 

r.eRorted on the*^SaPPll°8'^l°n* '^^^ ^^^^ ^^^^^ Vprocedure 

f^!lnVc|Lyed oTily aubmlttlng the doqurpenta, not croaa-checklng them With the 
:''appllcatlori| and as such was Intended to deter mlsreportlng. v 
<jj4m,. :■; ; ■ ■ -,: : , ' : ' ■ :) ■ . ■ , \ ' ^ 
Both'iriese procetiures were Implemented vpt the start of the achool year when 
meal appllcatllons are ; submitted and reviewed for. ,Bllglblllty. Thus i they are 
primarily deterrencevprocedureaa In most school dlatrlcts, .theae t;\y6 procedures 
were tested by schoonslylldjng personnel (e.g., principals and adhool/ecretarles) and 
Involy^tJ aU stbdenta. ' 



1 SkUUDI ll.3< 

* ■ ' ■ ' - 

Fl^ife*^ additional ■ quality -assurance procedures were Implemented later In the 
schoollfear after app He atlona had been approved. Thds- they .were primarily/, 
detection procediire3, f.ThB procedures were:, -^ 7 • - ' . > ' 

m. Documentt^onalstencv Check' '^ follo>Ing Approval fdr^-^ benefits to 
determine tfe document initially submitted iwith the application 




cocroboratSd' inform onvthe application. 



IV. D< %rnet)tatian^ with Conalatency Check to obtain 

do^Re^ of income and other >tjenef It * program eligibility 

i;/^^ed-^^ procedure is also referred to as 

iblj^-d^^dodumentation r^ . V 



V. - Parent' tBleDhonB ConferBncB^ ollowing approval of banefits to *rB-.dbtaln 
all application Itams in order to verify income and family plze Information 
originally reported on the application. • 

• yj. Locaftiird-Partv Checks (e.g., welfare office data exchange) foliowmg 
. J apto'oval of benefits to verify .^earned income or income maintenance 
4<-^. '^^gram'e^ljgibiU^ ■ 

''..VII^^ siate^^^-F'arty .bhBci<s . (i.e., computer wage tape match) following 
♦ approvajVof benefits, to verify earned Income or Income ^maintenance 
prograr^ eligibility iaf6rmation.\ ^ - < ^ 

J' Do^ument corfs^tency check, documentation after application with consistency 
check (or *olipw-iiBocumehtation), and parent telephone conference were primarily 
conducted b>8chM^uildio& officials. Local third-party checks were' conducted by 



^tetPlot pepflonnal and ate thlf^paHy ohioka Were oondMoted by 'the evaluation 
pontpaQlQP. All five Qf theae ppoQedur&V'n: qantraaE to the flrat t^^o proqedureb, 
vy«re teatad on a aarople' of atudentsi In efioh aphOQl. ' - 

_r .:■ ■•■■.I,,-,^, .^^.v/y.v ■, ' , , ^, ' . ■ 

; Th0 Implemontatlan and ovqIu^^^^ of tha quality^ eioour.anoe praoadurao wero^ 
: QQQbmpUahad ualng m axperlmantal d»Qlgni AoQordlng to thia daalgn^ cbntrol oohool 
dlatrlota we)ra matqh^d to axpeplmantql dtetrlota, ,Tha aavan quality aaaurenoa'. 
prdcedurea wera; then; ^randpnnly aaalgnad to adhoola within aaoh axparlmential 
district, A 'total of 16 3tiatB8--9 experimental and 8 dontrol--and a nationally 
j'epreoentatlva aample of ll4 aohool dlatrlota aartlplpa^e^^^ the pilot prdjeoti! 29 
e>^peplmBntdl altaa where the aeven quality aaaurance prq^adures were toat^d and'OS'^ 
control altaa« Ovarall,, tt:ie axperlm^^ntal ^altaa condlated of a total of 735 aohobl 
buUdlnga-KWlth a total enrollment of over 379,400 atudenta. Over 78,60Q IVPP 
applloatlona were prooeaaed, of - which 35 percent irioluded a document' with the ' 
applloatlpn,' Ten percent of tl^a appllcanta In the experlmerital altea wera aubjected 
to . one form o'f yerlfloatlon^ performed at the achool leval— parent telephone 
conference, follow-up documentation requeat with conalatency cheeky or document 
oqnsiatency cheek— and- either local or atate third-party cl:\eck /QT\ellglbIllty' for v 
income maintenance beneflta or earned IncdmjE 



; Jme procedurea were evaluated on the baaia of data from IVPP appllcatlona and 
non^VPp.. appilGayion3,- detailed outcome reporta-completed-by" the^achooi offlctala^ 
who actually conducted the, procedurea, queatlpninalrea completed by achqoL and 
diatrlct' peraonnBl, and In-home audita w^ a random aample of 1,810 appllcanta in 
15! experimental altea. Alao, queatlonnalrea were completed by each of the matched 
cpntrdl^ achpol diatricta. The reaultlng data were analyzed, land the reaulta^ are 
aummarized below. ' 



Study Cone luaiona / ■ \ 

• ^ Revjaed* Application la an Effective Deterrent to Miareportjnq . The 
rpviaed pilot atudy applicatibq reaulta in a aharply reduced etrwucgte of 
— 11^7 percent. Within thia per6entage, the moat aarious categoryXof 
_ mlareporting (i.e., appllcanta receiving benefita who are ineligible for free 
fneala) dropa to 1/2 of 1 percent aa a reault of the new applicationi. 
.further reductiona in thia error category aeem unlikisly. The 'rWiaed pildt 
atudy application doea^not preaent a diacoverable barrier to palE^icipatjon 
* for eligible houaeholda. ' 

Rev iaed Application l^eaulta in Low Dollar Errer Ldaa. When the ili7 
percent error rate ia tranalated into error coats (i.e., the percentage .of 
dollara lost to 'miarepoYting aa a function of total prggram expenditurea) 
the dollar error rate ia low — approximately 5 percent.'' ' 
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flppUoaUop crtv^^^^ thari l8 Qon6lnulog^9v|clenQB of r 

' aubBUantlarnMrnber QfpartlplpsntB ppeeWIng exoeas behsflts m^it a. 
5>3 p§rqint Qsae, error patp). '\ ' 

Suflta Inad faotlvaneBa .of Datftrreppg Unl<i]Qwii ^ It, Is apumed that the 

axp libit thpaarrffli^ for the deteppint 

' effaot of the rivleed^applloatlonv I iQwe^vepr bedauae tha pilot study was 
' of UmltBc) duration, It la not now po^plbla to Judge emplrlcsally the, 
, auotalned effeotlveneaa of the applloatlon, ever time. Study findlnga do 
> not pffer gujdanoe on how wldeapread or. Intanaa Inaqme verlfloatlon 
,< dataotlon prooedurea need to be. In order to kataln the, deterrent effaqt of 
j the Qpplloifltlon ovor time. ' \ , ~ - 

DocUmentatloh wltK> Application Ineffeotlve . NRequlrIng Inaomo or food 
' atamp eligibility idocumentatlon with applications did npt protiuoQ a 

ftlgnlfloant reduction In error. Moreovor, thl^ prooedujo radulted In. 

bai^lerB to participation for eligible houaahold^. V 

, Probloma Encountered Implementing Deteotlon Pro>3edureo; In pilot olteo, 
< even with extensive contractor training and teohnlqal. aaalstonce, aohool 
, cilstrlcta failed to follow through 'with full Implementation of aoveral 
detection proce.durea. Such 'compliance problema may be , encountered 
under normal operating .conditions. Study findings aqggBOt that tjils la 
partly due to the complexity of the verification procedurea, requisite 
.:.;jQllowup ajbtl^^ltles, &nd built-in administrative d la Incentives to cooperate..^ 

'• Detection Procedures Not -Cost'Effectlve . None of: the Income 
verification detection procedures save more money theg they coat.f 
Moreover, these costs accrue almost eixcluslvely to school\ districts and 
not to states or to the federal government. The high liS^al costs to 
conduct verification are due In large part to the fact thatXhlgher piald 
school personnel (e.g,, principals) typically conduct verification. 'In 
contrast to the revised application, the detectlop procedures, In general, 
present high barrier to participation for eligible households. \ 

Use of Error-Prone Model Makes Some Detection Procedur^' More 
Effective . Targeting appllGations for verification Using the error-prone 
model is more effective in locating errors than/random selectionV,^ The^ 
most effective detection .procedures were ; parent ^confeijencesx and 

' state-wage matches using the error-prone model. However^* ^ven When,* 
using the error-prone model, parent conferences were not bost-effective, 
at" least in the short run. The study did not produce measures of the 

* ^3t-effectlveness of state wage matches. ' 
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. INTRODUCTION 

In response to concerns about income misreporting and subsequent excess 
benefits in U.S. Department of Agriculture's (USDA) free and reduced-price school 
meal programs, Congress mandated provisions in Public. Law 97-35 to improve meal 
application procedures. This law specified a number of changes in the application, 
inc|uding the requirement that Social Security numbers be reported for all adult' 
members of applying households and that only reduced-price guidelines!/ be 
distributed to applicants. U«'5. Department of Agriculture was empowered to 
require applicants to provide proof of eligibility and to require School Food 
Authorities (SFAs) to verify eligibility. In addition, the Secretary of Agriculture 
was authorized to conduct a pilot study to verify data on « sample applications. 
USDA initiated the Incomf Verification Pilot Project (IVPP) in response to that 
authorization. , 

1.1 INCOME VERIFICATION PILOT PROJECT , 

The Income Verification Pilot Project developed and tested a variety of quality 
assurance procedures to prevent and detect the misreporting of income and family 
size information on the school meal application.2/ The following major research . 
issues were addressed: ' ^ 

m determine the capability of the quality assurance procedures to reduce 
error rates by deterring and detecting applicant misreporting; 

• . determine if it was feasible for school districts to conduct the quality 

assurance procedures; 

• measure the costs to perform and savings that result from each quality 
assurance procedure; and. 



• ' examina the barrier effects, if any, of the procedures on program 
participants and their families*. » ' . 

The pilot projfct was conducted during school years 1981-82 (Phase I) and 
1982-83 (Phase II). * Phas'e I Involved a geographically limited sample of 13 school 
districts Aid assessed the effectiveness of a revised meal benefit application as well 
as additional explanatory information to reduce applicant misreporting and error. In 
contrast. Phase II Was a large-scale test involving a total of 120 experimental and 
control school districts throughout the country^ In this ph^se, a scientific 
experimental design was used to test the effectiveness of a redesigned school meal 
v^benefit appUpatlon and six-other quality assurance procedures designed to deter and 

detect error^ in reporting income and household size, , 

■ • . . . ' . ■ ■ ■ 

V/ ■ . ' ■ ■ ■ ■ 

1.2 th;5 and other reports of the;^rroject's findings 

This is the fourtjh repqrt from the Income- Verification Pilot Project (IVPP), 
Eap^ier reports examined specific Issues relating to legislative effects and 
rdisreporting based largely on data collected in the 1981-82 school year (Phase 
/ I)«2/ This report presents the results of a comprehensive assessment of ^specific 
quality assurance procedures tested^in a national sample of school district^ in the 
1982-83 school ye^r (Phase II), ^ 

Although the pilot study procedures resemble Income verification procedures 
. recommended In Interim regulations by USDA, the pilot project began before the 
regulations were implementediL/ While this report is not intended to provide an 
assessment of the USDA's recommended procedures, the findings reported here do 
provide Information about the potential Impact of some of USDA*s interim 
verification regulations. 

The remainder of this report presents major findings ^^om phase II of the pilot, 
project. Chapter 2 explains the "various quality assurance procedures tested, and 
Chapters 3 and 4 present research design and data collection. Chapter 5 presents 
the findings on the feasibility and. effects of the free and reduced-price ^ool meal 
. application developed specifically for use in the IVPP project. Finding' on the 
feasibility and* effectsf 3f requiring income documentation with the free and 
reduced-price school mell application are reported in Chapter 6, Chapter 7 
presents the findings on the feasibility and effects of quality assurance procedures 
that take place after school meal applications ar§ processed. The results and 
conclusions of the pilot project are. summarized in Chapter 8, 
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ENDNOTES 



1/ Previously both free and reduced price guidelines were distributed. ' , 

2/ Seven quality assurance procedures were tested by the IVPP, Chapter 2 
describes each procedure in detail. 

■ ■ ■ , ' ■ 

3/ Earlier reports include: - 

Income Verification Pilot Projedt; Findings on School Meal Participation and 
Legislative Impactt 1981-82 School Year. Applied Management Sciences, 
Silver Spring, MD, December 1982, . 

Income Verification Pilot Project; School Year 1981>82 In-Home Audit 
Findings, Applied Management Sciences, Sliver Spring, MD> Apriri983. 

. ' • ' ■ ' • •' ^ 

Income Verification Pilot Project; Development of an Errofr-Prone M odel for 
School Meal Programs. Applied Management Sciences, Silver Spririg, MD, 
August 1983. / ^ . , ' 



• 

4/ 7 CFR, Part ^45. 
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THE-QUALITY ASSURANCE PROCEDURES ' i 

.\ The effectiveness of a redesigned meal benefit application and sj^llfferent 
quality assurance procedures was tested iDy^ff^e Income Verification Project 
(IVPP). Most school districts selected to participate as sites in the study tes^S^all 
prbcedures. In accordance with the experimental de^igni however, different{8chool3 
in each school district implemeni{;ed and tfested different combinations of 
p|*ocedures. No school tested more;, than three .procedures, anfd in somq instances 
they tested only , one or two. This chapter - describes each quality assurance, 
procedure, the rationale rdrHQcluding^it in the national experiment, and hoW;.it was 
expected to operate.l/ ■- " / . 

. . \ ■ ■' - ■ ■/// . 

1.1 BACKGROUND AND DEVELOPMENT ' * / ^ ^ 

Quality assurance procedures kre^ implemented to prevent and to detect errors 
that result in the inappropriate expenditure of -funds. Quality assuj^ance procedures 
are used to prevent and detect misreportlng by appllcants^and, erriOrs caused by 
administrative agencies^ IVPP focused only oci thfe prevention , and detection of 
misreportlng by applicants, and did not examine adminjjstrgative errors. ,tv1any federal 
' agencies responsible for benefit programs, Including Aid to Families with Dependent 
Children (AFDC), Food Stamps, Supplemental Socia^ Security, 'and Student J='inancial 
Aid, use a variety Of quality assurance procedures to reduce fraud and abuse by 
beneficiaries. Quality assurance in these programs is designed to assure that benefit 
awards are made? on the basis of correct information. Other agencies such as the 
Internal ''Revenue Service use similar procedures to ensure that the government 
collects the tax revenues each citizen is required to pay. 



\ Uniike^hese prograrnsr the USD^^^ 
have only recently begun to address the problem of income misreporting and excels? 
benefits.^Jhere are a numger of reasons for t > 

. v ' , . ■ ■ • ■ • * ' .' ' " ' 

• Low Benefit Amounl:, .The school meal programs have a relatively low 

. per-student dollar benefit rate— approximately $200 per student annuall)^ y 
which is approximately 10 percent of tba^aver age annua^AFDC payment ^ 
(federal and state funds) for one child. Efforts to reduce .misreporting are 
co^t-affeative onfy if the amount saved through prevention and detection 
of misreporting is great enough to offset the additional costs associated ^ 

; with quality, assurance. / * ^ 

• Historical Reliance on Self Certification by Applicant. Traditionally, the 
school meal programs have been permitted to accept' self-reported, 
unverified Information about household size, and income art d have based 
eligibility determination on jfhese reports. Also, prior^to the 1981 passigge, 

^ of Public Law 97-35^ 'school districts, were only permitted to verify 
app^^atiqns on a .'for causeV basis. Ip contrast, many other federal 
Assistance prograr^is require applicants to submit documents that verify 
>hdfusehold incomiB the benefit application. Because sdhdol 

-^districts are accustbme'd to -a Relatively simple application process, the 
implementation, of quality assurance in the forgi of ^ verification 
requirements jmay initially meet resistance from both parentis and sdihdol 
district officials. ' . * / , ;// "^z * 

• - Lack of Specialized PersonneL Applications for meal benefits are ^ ^ 

processed by SFA^or other school off iciials who have many other\ v 
responsibilities and limited time, to' devote to t^he eligibility detergiination , M 
process. Implementing quality assurance requirements would place ah ^ 
^ « .additional burden on these personnel, fn contrast, otwer federal assistance 
programs have specially trained staff t^profie^ • 
quality assurance procedures. \( ^ ' ' J 

t Absence of Federal Funds Earmarked' fdc>buallty Assurance. School food, * <. 
authorities' do not receive any direct federal funding to^^onduct eligibility^ 
determination and quality assurance efforts.2/ other federal programs ^ 
provide funding and other l^ricentives to. support, quality Quality 
; r assurance may introduce ^ addittbnal ; staff requirements that are not 
currently funded, thus inhibiting the ability of school districts to comply 
. . effectively with verification requirements. < ' ' 

' • Limited Time^ for^ Application Processing , Families are required^ to ^ 
reapply annually for meal benefits. Because schools traditionally^^ 
A . . complete administrative paperwork at the begiinning of. the schoolyyear,* ' , 
virtualjfy all meal applications are processe^^efore October^af each 
school year. In other federal ^ 'assisj^^^/programrf, eligibility ' • 
determinatiori^nd verification occur routinel^^foughout the year. 

.... ■ .■■ : • ■ :y • . ■ .T .' ■ • 
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^ To|i)tpjentify^ a%jranpe prqcedur^th^t mig^ the' 

3chool*feeal p applicatiorr^process, two activities were undertaken. First, 

qualityl/aasurancer techniques already used in other federally-supported benefit and . 
asaistahqe ?pr6^^^^ were examined. Second, quality assurance proceduresvthat have" 
been piri^yibiisly initiated by very few school districts were examined. The research 
identjfifd^^^^ programs that h^d implerppnted quality assurance 

procedures'.as part.of either eligibility determination or a quality control process. 

l:ilPi;rr' ^' , . V • ■"• ■ .'■ ■■• .. 

FoufVm^j^t categories of qu 



^ f^lji^j^:' Incomg - Documentation . Written or verbal jhformation wais used to 
^ confirm applicarit's earned and/or unearned income. In virtually all^casesj^ 

llpl^ Vthe original source of the docurnent ob^aihed by the applicant or)recipient 
'was another agency or indiyidtial (e.g., employer), . ' ' v 

J/^.^ \ Piggybacking on Anotlhep Program's Verification orV ^gfiqibility 



Determination Process* Incdme amounts previousjiy verified ^y another 

^m^b ben 



r 



program or certified eligibility to receive .that program^b benefits are 
soughti^ It is also possible to* request the prograggj^ verify appli(;}ation 
, information as part^^ of its qualit^ assurance procedures. 

• ' Computer File Matching . Income and household siize inform aUprf^on the 
application wa? compared to information in the computer^ ffTw of oth^^ 
, assistance pragrams to cortfirm eligibility or identify discrepancies, ' * / 

■■■■■■ ■ I ' ■ V 

Error'^Prone'' Profiling. Results of a statistical .analysis that provides a 
Vprofile of applications most likely to be in erroj were used, -This 
statfistical profile was then jjsed to identify applicafiions with a high- 
likelihood of errorfor verification* ' J 

Typical methqd^Jsed. in school district^ that had adopted income verification 
prior to IVPl^ included: (1) requests for documentation from all applicants when 
• meal; benefit applications were submitted, (2) verification after the application was 
approved on a random sarppls of ^applications, through teleptjo'ne conferences with 
parents, o^ (3) contacts with employers or public assistance agencies to confirm the 

; ■ income of a sample of applications. |- ^ 

} ■ ■ ■ J* . ■ ' ■ . 

Knowledge of quality assurance procedures used by other agencies provided the 

basis for ^specifying a' range of potential procedures to^ deter or detect applicant 

misreporting. However^ many of these procedures are very complex and difficult to 

implement without' a significant' experience (base. In selecting the specific 

prob^^ures for testing in the pilot project, consideration was ^iven to school 

operatihg> characteristics noted e^lier and compatibilityVwith the timing of 

, appllcatjian-f'elated activities 4n the school n*eafproigrams/ For example, virtually 

jail households ^apply ^r free and reduced-price meal ben^efits at the start 'of each 

sch"^, year.2/ School vOr^ SFA ^ffici^ls determine eligibility based on the 
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' appilcation. Quality assurance, when it is performed, usually ^occurs after:'eligibility 
hdb been deterfnihed and benefits have been awarded. This sequence, of activities' is 
' follow^ to avoid delays in providing meal benefits at the gtart of the school year 



when the , bulk of applications are. received and^procesfsed, Jfi^^ it was necessary to • 
select dnd' mdciify existing quality; assurance' prd(cedures t^^ilhqr^ 
. * ^r use by school ydlstricts. • F^examf^^ benefit prdg,rams require that 
applicants provide documentation of all incbme. vObtlining and reviewing these 
do^uments is very, labor intensive,* Tq^ actgmmodate^^^^ 'School^ 7 

districts, this requirement viae modified -for IVPP.;t^include onLy docurfientatioh of 
the primary source of Kdusehold income. Slnri|llif simpUfications and adjustments 

■ . «' ■ ', , ■. , * - ' ' ■ ^ 

(. Were made in all of the quality.assurari^^p; 



icijdui^;tested in IVPP. 



"Uifimat'ely, b' me^l:appli<fBiiqn:Wim for the IVPP and six., 

quality^^urance prflcedjuriSs w^^Jl^j^ignedt^^^ assurance 
procedures are listed be 
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II. 



in. 



liSl^^ .Application) / This procedure was 

designis^d fe^^^ data needed to support the ) 

legal aff^ veHfication activities. 

J^l^cument^^io^^ Jhis procedure was designed .\ 

fe^ deter miswpojii^^ requiring information abofiit earned incorrife or;^. 

other ^hj9ii6f it p^^^ eligibility (e.jg*, food 'stamps! This procedure 

^ involved o^ /^ with the application^ not 

' cross-dheckino^ the^'" document ' with ^ corresponding j 

infbrmatltilii^i^^ - s * ' v ' . ' 

Document CdiiBisfency Check. This procedure was perfbifmed on a sample 
basis to detect errpr only for applicants ^ Who were required to complete ' 
Quality; Assu^^ !!• It Wk place following approval for 

bBn^fits:^tli determ th3 docgment{^ submitted Avith the applipation » 
corroborated information on the applicationi. 



IV. 



V. 



VI. 



V 



VII. 



u^ocumentation* After Application ^with .Consistency Check . This ^ 
procedure ^as also performed to detect error? and used a\ sample of 
applicants .who 'di4 nqt .undergo Quality Assurance Procedure I. , »Tri§^ 
sampled: housafiold was req to . provide documehtapy (DPoof to 
corroborate information on the application aftgr the application was 
approved.; / ^ , \ ^ ' 

Parent Telephone Conference. This procedure was perform e(|^ on a sample 
basis to detect error following approval for benefits. It was designed to v 
verify income .and family size information on the application by >; 
re-eliciting this informatiori from the applicant. * 

Local ?Thlrd--Party Contact (Interagency Data Exchjnq^ t Following ^ 
approval for benefits, the Income maintenance pp^ram eligibility or 
earned Income of a sample of applicants was verifle^Ttpydetect, error. 

State Thlrd'-Party> Contact (Interagency Tape Matcher 
for benefits, the ^ earnf^d Income or. Income mal 
eligibility of a^ sample of applicants was also vii'erifled to detect error. 
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the first and second procedures wjsre jmplemented during the application process: 
applicants were ' required to comply with the procedure or the meal applicatlori 
would not b^i approved. The remaining procedures were implemented by school/ 
officials later in the school year. Each procedure was designed so; it could be 
implemented by staff from either the school building or school district. The choice 
was left to the individual school districts participating in the pilot project to 
minimize disruption of theic administrative structures caused by implementation of 
the proCedufes. 

2.2 DESCRIPTION OF QUAUTY ASSURANCE pKoCEDURES 

The rationale for eaclVpf the seven procedures and a description of how each 
was intended to operate^i^^li^ IVPP 

2.2.1 Quality Assurance Procedure It Redesigned Application Form 

To deterr misreporting and to support the analytic needs of the pilot project, a 
completely revised meal benefit appllcatign was developed. This ap|3lication is 
referred to as the^IVPP application in the remainder of the report, and a facsimile' 
copy is provided in Appehdix A. The application was designed to enhance reporting 
accuracy and to provide necessary informaf:ion and authorization for other quality 
assurance procedures. It should be noted that there* is* no standard application form 
for school meal benefits that all school districts must use. The Food and Nutrition 
Service of USDA Issues regulations concrerning the items that must be Included on 
the application, and develops a model application form and acpqmpanylng model 
parent letters. States and districts then use these model applications and parent 
letters as the Basis for their own foriris. 

The form incorporated features to reduce specific types of misreporting 
revealed" in analyses of^ reporting errors, using Phase I applicant audit data. The 
redesigned IVPP application varied from the application form recommended by FNS 
* for the 1982-83 school year (and used by most school districts) in the following ways: 

• Income Detail. It requested applicants to list aH sources of income 
separately for each adult member of the household. The income reporting 
^ ' section was expanded because the analysis of Phase I applicant audit data 
revealed that failure to report' wages from all jobs and for all adults was a 
common source of misreporting that led to the award of excess benefits. 



i Ver^ Waiver* " Under provisions of state and federal prlvsicy 

. legislation, state and local agenci&s ma/ 'not release information without 
specific per/nission from the Involved individual. Therefore a waiver was 
^ ■ required for each adult whose income and/or eligibility for an income 
maintenance prqgrarn wouW collateral, contact. 

Furthermore, requesting each adult to sign the waiver, was expected to be 
a deterrent because applicant^ would clearly understand that the accuracy 
of the information they supplied was subject to verification. 

• Descfiption of Legal Basis for Information Requirements. ' The IVPP 
application provided a complete description of applicable provisions of the 
Privacy Act^ the legislative basis for requiring the information sought on 
the application, and the consequences for not providing 
required/requested information. (In'additioh^ it provided guidance for 
calculating monthly income.) * 

The pilot project tested several features of the revised application: (1) its 
at^ility to discourage (deter) misreporting; (2) its administrative feasibility; arid p) 
ita^effects, -if any, on discouraging^;,(^ eligible individuals from applying for 

benefits. Another purpose of the pilot project was to obtain information nefeded to 
support subsequent verification activities. ' 



School districts that used the IVPP application received English and Spanish 
language versions.^/ Districts were instructed to follow their usual procedures 
for distributing the revised applications^ . Consequently, in some districts, 
distribution and prqcessing of applications was handled at the district level; in other 
dlstricts/schpol building personnel performed these tasks. 

Processing required re^viewers to determine both the completeness of the 
application and whether the household was eligible for school meal benefits. A 
complete application was defined as one that included the signature of an adult 
household member, thd^ Social Security numbers for all adult household 
jpnembers,!/ and information about household income and size for the complete 
month prior to completion of the application.^/ If any of these four items was 
missing, the application was considered ihcomplete. Incomplete applications were 
returned' to parents with a form explaining what additional information was 
required. Eiigibility was not determined until a complete application was received. 
If the household did not resubmit the application, a meal benefit determination was 
not made. . . 
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' 2t2»2: Qualijby A88^^ at the Time of j ; 

Application ;?/ ■ ' . / •: 

\v^ : In thte applicants were requirpd to enclose with lihe completed 

• application a document that verified the primary source^ of income fd(r each adult in 
the household for the month prior to the submission of the agency'^ application. 
This time period was selected because it coincided with the agency'sldefinitlon^f 
current income used to determine eligibility. To comply with the Requirement, 
applicants could pfovide' a document confirming eligibility .for food stamps. Adults 
with wage income were given- the option of documenting their wages. \ Adults who 
• had no wag^s were asked to document income maintenance benefits (e.^L AFDC) or 
other primary sources, of income. See Appendix A for a copy ofMnstructions 
y regarding documentation for applicants. The focus on documenting wages was a 
result of the Phase F data, which indicated that earned; income was the major source 
of misreporting and thereby contributed to the improper award bf meal benefits* 

School or SF A personnel were instructed, not to process the application unles/ 
appropriate Income documentation for e^h - adult was submitted|* Where 
documentary support, was missing, applications were returned to the househald with 
an explanation of what types of supporting documentation^would be acceptable. 
This procedure had the potential to delay eligibility determination, pnce* the 
application and supporting documentation were received, school officials/ were 
'instructed to determine eligibility based solely on the information contained In the 
application. They were instructed not to compare Income lnformation\ on the 
application to the documentation but rather to check the 'documentation ^nly to 
ensure tbat it covered the proper date, the proper in'^ivldual(s), and acceptable 
f evidence. This check could be performed quickly and did. not -require exten^iye 
review,'rBcomputatlon, or follow-up. ■ ■ ^ ' 

The procedure involving documentation "at the time oj application was tested 
because it placed the burden of proof on the applicant rather than the school district 
and because it could be verified at the time of application. This procedure 
resembled those used by other benefit programs that apply a "means test**, to 
determine eligibility. It did not require school^istricts to develop sophlaticated 
dataVor administrative systems because the applicant provided the docuhnent. It was 
.. a compromise between requesting and thoroughly reviewing^ (Mk^ at| the 

time of application and requesting documentation only after the application had 
been, approved. The procedure; was expected to both deter fraudulent behavior and 
permit subsequent detection of misreporting. ^ . 

. ■ -. v, . . r ■■. v '-r ■ ■ . 



Quality Assurance Procedures'"! and If were conducted as an Integral part of, the 
apiilication process. The five remaining pr6cedupeg''wi^^ after eligibility 

for meal benefits- had. bgen determined and rnvohiea only a sample of approyed 
applications, ^ . , ' , * 




2*2,3 Quality^ Assurance Procedure Illr Document Consistency Chec1< 

Under this procedure, the determining official compared the income amount 
indicated on the application with the corresponding amount(s) indicated on the^ 
docament(s) accompanying the application.^ Akernatiyely/ recipients could have 
confirmed their eligiblity based on a proxy for need such as evidence of eligibilitV 
for food stamps. Where the documentation distagreed with information on the 
application, officials recalculated : eligibility using the Information from the- 
documentation. . If the discrepancy was large enough to affect eligibiljity status, 
households were notified by mall arfd asked to explain the difference. If the 
applicant failed to respond or was unable to ^expldip adequately the difference^ 
benefits were changed to reflect eligibility status based, on the dbcument(s). The 
procedure; was condubt^d using a sample of applicants who subtriitted documents 
with the applications; • . . ^ " ' ^ 

The document consistency check is a verification procedure intended to detect 
errors'.reported on the application. This procedjjre was .tested because it was 
considered a {tactical and relatively simple method by which school personnel could 
validate«^applicant-prpvlded information for a sample of households. The document 
consistency; check was performed after appli.cants were approved to avoid potential 

delays.;in the annual eligibility determination period set forth in program regulations. • 

r • ^ ■ ■ . ■' ' " *. ' ' ' ■ ■ " ■' ■ ■ ■' '• ■• 

%AA Quality Assurance Procedure IV; Documentation After Application ' 
with Consistency Cjneck ^ * r 

In^^ofitrast to requesting documentatibn'with the application (Procedure II), this 
procedure was implemented after an application for school meals had been 
completed and approved. A random samnple of approyed applicants was contacted in 
^^i^riting by the determining officials and required to submit documentary evidence of - 
what their income was for the mc^th prior to application; Recipients, were given a 
list of acceptable forms of dd^mentatlonV This' list was similar to the list of 
documents used \x\ Quality Assurance Procedure II. Adult household members with 
wage income were required to document their earnings. Those with no wage income 
were required to document income maintenance benefits ^such- as AFDC or food 
stamps (if any) or other primary sources of income. - ' , ' , 

12; , ■ 



/ Upon receiving documentation, school or SFA officials reviewed the materials 
as in Quality Assurance Procedure in to be ^certain that information for all adults 
with reported income was included, that documents covered the month prior to. 
application, and that the documents were acceptable forms of evidence. If the 
documentatio*n was incomplete, recipients were notified by mail and. told what 
additional information was needed. If the; recipient failed to responS", meal benefits 
were terminated. If documentation was not consistent with the information on the 
application, officials recalculated eligibility using information on the 
documentation. If the discrepancy was large enougl;i to affect eligibility, the ' 
household was contacted and asked to explain the difference. If the household did 
riot respond or could not adequately explain the discrepancy, eligibility status was 
altered leased on the new household income indicated by the documents. 

Documentation after application with cpnsistency check 3vas tested because it 
was a method of detecting error. The procedure placed the burden of proof on the 
applicant and did not requira computers to confirm the accuracy of 
applicant-provided information. Because the request for documentation was made 
after the application had been approved, it did not delay the receipt of meal 
bi^riiefits. , 

Like documentation with application, the procedure tested whether households 
maintained records and whether employers and income maintenance offices were 
willing to - supply xecords to the households. Th^ procedure differed from 
documentation* with application in that it tested whether recipients would cooperate 
with a quality assyrance procedure implemented after benefits began. Tinally, the 
procedure provided an error detection and correction capability by identifying cases 

of misrepbrting. . * " . 

^ ^ ^ ; ■ 

2.2.5 Quality Assurance Procedure V: Parent Telephone Conference 

This, procedure was also implemented after applications were reviewed and 
approved. School district officials contacted a random sample of approved 
households by telephone to determine household size and Income in the month prior 
to submitting the application. The information 'obtained over the telephone was 
compared to that on the application. Where discrepancies were found, eligibility 
was recomputed based on the corrected information. Households were then asked to 
explain the difference. If no. acceptable explanation was offered, eligibility status 
was changed on the basis of revised household income or household size. Parents 



were notified in writing of the correction and Its effect on eligibility* If households 
could not be contacted by telephone! a letter was sent instructing the parent to 
contact the school official. . . ^ , - 

Parent telephone con ferepces were a detection'^rocedure designed to identify 
' intentional or unintentional errors in reporting household size or iocome. The 
procedure was tested because it was>a low^-costi low*burden method of verifying 
information on the application. - ^ 

2.2.6 Quality Assurancg- Procedure VI; Local Third-^Party Contact 

Local third-party contact /is an information exchange betweein the 'school or 
SFA and the loca],Food Stamp and/or Public Assistance office. For a sample of 
meal recipients, the local social services office was asked to indicate whether each 
. recipient in the sample had received benefits (and^ the amount of the benefit in 
instances where AFDC was verified) for the month prior to submitting the 



"'^>-aeplicatidn.§/ School officials compared this information with application data. 
If discrepancies affecting eligibility were found, parents were notified. Where the 
discrepancy bould not be explained, eligibility status was redetermined based on^the 
third-party information, and benefits were adjusted accordingly. ' 

This .procedure was designed to detect misreporting without the direct 
involvement of the household. This procedure was selected for testing because it 
relied on an independent^durce of information and eliminated the opportunity for 



applicants to alter documents. Further, a relatively low requirement of school 
personnel time was expected. Finally,* tha prodeS^^^^ 

recipient only when the third-party 1 provided information that disagreed with 
infoi^mation on the application. - 

Phase I of the Income Verification Pilot Project indicated that households who 
receive public assistance or food stamps typicaU^ have incomes low enough to 
cjualify for free school meals; therefore, this proce^eice^pj^s not expected to identify 
many households ineligible for benefits. However, USDA's Office of the Inspector 
^^^pineral recommended testing the feasibility of local third-party data exchange^ as 
a method of verifying application information. F*urther, knowing how many 
households received benefits (I.e., food stamps) they did not report on the 
application was seen as useful for designing future application procedures and 
instructions. The total household income was not verified by these agencies because 
of differences in the way income and household are defined by the school meal 
programs, food stamps, and AFDC Jl^ 
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2.2.7 Quality Asaurance Procedure VH: State Thlrd^Partv Contac^^^^^ - — 

' The' seventh and last procedur0,,,yi/aa i^ in states with statewide 

computerized wage, food stamp, and/or public assistance Informatioh s^ystems and 

^j^^the capability for computerized data excharigea. The names and Social Security 
' numbers of a sample of household members who signed a waiver permitting 

. collateral contact were forwarded to the appropriate state agencies.lQ/ State 
employment security offices provided information about wage Income for the 
quarter including the application month for each listed ac|ult household. member. 

^ Public assistance agencies reported receipt of public assistance and the dollar value 
for the month prior , to application. Food stamp offices confirmed receipt of 
benefits for the^^^r^^^ V 

information was returned to t^ pilot project evaluation . contractor who 
identified cases with discrepancies between the information on the data tape and 
the applica^^ discrepancies were found, eligibility was recomputed based 

/ on the data tape information to determine if eligibility status would be affected. 
' - .Because definitions of total household, income differ by program and because this 
' was only a test of the feasibility of automated, state-level tap0. matching, results 
, were not shared with the schools or ^F" As. Meal benefits were not corrected as' a 
result of the findings from the state-level collateral contacts; Rather, the results 
were used analytically to investigate the, error detection capability of the 
procedure, and not the feasibility or costs^/' • . 

' Tape matching has been used to verify Income information In a variety of 
i federal assistance program^, including Student Financial Aid, AFDC, Food Stamps, 
and Housing Assistance. The procedure wai% specifically suggested for examination 
by Office of Management and Budget and USDA's Office of the Inspector General as 
a potentially effective method to detect misreporting in the school meal programs. 
It was tested as part of IVPP because the method was expected to be a very 
accurate way to detect unreported and underreported income. The procedure relied 
on an independent source of information and did not present the Verified household 
with the opportunity to alter or withhold evidence.- In addition, the procedure was 
expected to reduce the burden on school officials and third-party agencies by 
centralizing commiinicatlons arid automating the matching* process. 
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2.3 IMPLEMENT ATIQN OF THE QUALITY ASSURANCE PROCEDURES 

TheJVyp evaluation contractor prbvldecl technical support to school districts 
for all Phase II procedures. The contractor supplied copies of the IVPP application 
and madel letters \that could be reproduced and sent to households explaining 
application requirements, used to train SFA and school officials in the pilot 
procedures for processing applications andr verifying, information, and used to assist ] 
all processing and verification . tasks* In addition, to assure random selection of 
recipients and conformity with the sampling and experimental designs, the 
evaluation staff drew samples or controlled selections for all verification activities. 

It should also be^noted that the participating experimental school districts were 
instructed to apply appropriate sanctions when applicants failed to comply with 
application OP quality assurance requirements and when verification revealed the 
award of excess benefit/ Applicant failure to comply resulted in termination of all 
meal benefits^i^/ Finallyj, if verification revealed excess benefits, benefits were 
adjusted to the appropriate level, i.e., reduced or- full price. In all instances, 
applicants and recipients were permitted td file a formal appeal with the school 
district in accordance with regulations set forth in 7 CFR 245. 
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ENDNOTES 

1/ Ghapter 3 explains the atudy dBsign and how school dlatxlcts and jndividgal 
~ schools were assigned to •experimental groups for the purpose of testing the 
procedures. ' t , ' 

2/ State agencies responsible for program administration do receive State 
' Administrative Expense (SAE) funds and . are allowed to use these funds for 
quality assurance, if done at the State lev^l. 

3/ Applications from families enrolling children later In the school year are 
~ processed when the children are enrolled, - 

4/ Applications were also translated Into Vietnamese and two Chinese dialects. 
5/ Adults were defined as persons 21^year3 of age or older. 

6/ Program regulations require the use pf current Income to determine eligibility, 
" which is defined as the most recent complete month prior to the month the 
application la submitted. 

7/ Documentation of Income as used In this report refers to an official document 
prepared by the income or benefit source (e.g., employer or social services 
agency); This use differs from 7 CFR 245, which uses the term documentation 
' to refer to minimum reporting requirements for an application to be considered 
complete. 

8/ In most instances the food stamp and AFDC programs were administered by the 
same local agency. 

9/ The school meal programs only consider Income that could be used to pay for 
meals while other programs also consider the value of; savings accounts and 
assets such as proper tyw 

10/ This waiver was Incorporated In the IVPP application. 

.. . ^ . , " '■' ' ^ ' . .. ■ y ' ' \,' ■ - ' . - ■ 

n/ All other procedures were evaluated In terms of effectiveness, feasibility, and 
costs. - 

12/ Except for Procedure VII, State third-party contacts, which was conducted by* 
the evaluation contractor, not the schooLdlstricts, 

■-■/ . ■■■■■■ ■ ■ ■ ■ ■ 



-RESeARCH DESIGN 

The design used to test tha seven quality assurance procedures had two primary 
components: a sample design for' selecting a national sample of school districts 
through which the quality assurince proceidures would be tested and an experimental 
design for testirtg the procedures within the selected school districts. 

5.1 SAMPLE DESIGN ^ 

A four-step samplel design was used to select a national sample of school 
districts included in the experiment. 

Step 1 " iSelectlon of States ; 

Fourteen states were initially sampled--seven experimental states^ and seven 
matched control states. The states were sampled in matched pairs on the basis of a 
lattice sampling design that ensured representativeness in terms of state population 
.and geographic region. Because state governments have regulatory responsibility 
for local school districts, state government cooperation had to be obtained before 
demonstrations could be established in school districts. Of the seven experimental 
states initially selected, two declined to cooperate. The two refusing states were 
replaced by three to guard against additional refusals. The final design included 
eight experimental states apd eigFt; matched control states. \ 

Because study involvement in the 'matched control states was limited to a mail 
questionnaire of sampled SFAs, state cooperation was essential but not required. 
However/ control state officials were notified of the study. 



3J,2 Step 2 ^.Selection of Experimental School DlBtrlcta V 

Within the eight experimental states, five school dlstricta per state were 
sampled. The sampling frame for experimental school districts was restricted to 
public scliool districts participating In the^ school lunch program whose enrollment 
spanned a,K or 1 through 12 grade range. Private, schools and public school districts ^ 
with a restricted grade range were excluded. The school districts were stratified by 
total enrollment, and sampled, with probabilities proportional to enrollment. For 
•example, a school district with ah enrollment of 10,000 had twice the probability of 
selelctjon ot a school district with an enrollment of 5,000. In the experimental 
states, 30 of the 40 sampled school districts agreed to cooperate. 

Because of the nearness of the start of the school year^ In one of the 
replacement'^ states, It was necessary to modify procedures. Three experimental 
school districts had already committed resources to a non-IVPP application. 
Therefore a modified set of procedures was adopted in those sites that used the 
' ♦ local application forms and did not include the requirement for documentation with 
application (Quality Assurance Procedure ID.I/ To assure that this modified set 
of proc^ures could be independently .analyzed, school districts with large 
enrollments were oversampled. 

3.1.3 Step 3 - Selection of Control Districts . 

Within the experimental states^ a matched samplp of 30 control school dlstrifcts. 
was selected. Each cooperating experimental school district was matched to a 
withln-state control school district on the basts of enrollment, enrollment 
percentage in poverty (as a proxy for program participation), and urbanicity. The 
withln-state control school districts were requested not to conduct any verification 
eff orts and to complete a mail questionnaire near the end of school year. AH agreed. 

3.1.4 Step 4 > Selection of Naturallstife Comparison Group 

In the eight control states, another matched sample of control school districts 
was selected. Each cooperating experimental school district was matched to two 
school districts in the matched control state. Matchinti| was accomplished on, the ; 
basis of enrollment, enrollment percentage in poverty, and urbanization. The 
control state matching school districts provided a naturalistic comparison group. 
Therefore no requests.were made to conduct or not to conduct verification efforts. 
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Sadh/of^bhe QohtrQl ata,te QqliQQidlQtriQta waa natlfiad that It would be (noluded In q 
mall eur^^^^^ be oonduqted later la flhdy/a the 

geographic diatrlbutlon'of the, aample^ experimental and cantrol dlotrlota. 



3t2 EXPERIMENTAU DESIGN ,> ^ ^ 

Within the experlf]gental school dlstrlUtSi every .school building waa ranqbrnly 
aaalgned to matched treatment groupev Exhibit 3^2 dlsplaya^ the school treatm^^nt 
group assignment method uaed for the wjthin-achool district. School troatmcint 
assignment involved four stepsi . / 

3,2il Step 1 > Asaignment^ to Quality Assurance Procedures I and 11 

Schools were statistically matched and then randomly asslgnejd to 5roup 
Groiip B. The schools were statistically ^ matched In terms of grkpe 
(elementaryy junior high, high school, special) ^and enrollment. ReferrlngCs 
3.2,. Group A contained dne-thlrd of the schools In a district and Group B 
two-thirds. Both groups used the IVPP application (Quality Assurance Rro'pBc 
However, schools in Group A required tha[t the documentation atcomfiany 
application (Quality, Assurance Procedure II), whereas schools In Group B dldtfot. ; 

3.2.2 Step 2 - Assignment to Quality Assurance Procedures III and V ^ ;.; ' / 

Group A was subdivided Into tWo matched groups of equal size, Al and A2/ I 
Group Al, the evaluation contractor drew a 10 percent random sample 
applications that school personnel then subjected to a document atlt^L'oQnsiste .^ 
check (Quality Assurance Procedure III). In Group; A2, the contra^jr drey a 10 
percent random sample of applications with* whom school offlcial^;:!^ a 
parent telephone conference (Quality Assurance Procedure V). 






3.2.3 Step 3 ^ Assignment to Quality Assurance Procedures IV and V 

Group' B was subdivided into two matched groups of equal slz^ Bl and B2. In 
Group Bl, the evaluation contractor drew a 10 percent random sample of 
applications. School officials then conducte"^ a parent telephone conference with 
sample applicants to verify application information (Quality Assurance Procedure 
V). In Group B2, the contractor drew a 10 percent random sample of applicants. 
Applicants in the sample were required to comply with the requirements of the 
documentation after application with document consistency check (follow-up 
documentation) procedure (Quality Assurance Procedure. IV). 

■ '20 ■ 
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In all sehQQls, the OQntriQfeop dp^^^ a leeand 10 pspsi'nt p9n,«jQm sampls of - 
WllQsntSi', Th§ ieqond flample ponaNtid Qf ^ perqent from the firat aample an"^ 5 
. percent not In the arlalnal aamplei AppHeatlona In thi aesonfl sample were subjeot 
to state or loael thlrd-perty Qheoksi LqqsI third-party Qhaol<§ ((Duality Aaauranae 
n Prooedure VI) OP itata-leyel oheoks (Quality Aaawranqe RpQQedure VII)^ were 
(jQndMQted depending on whether arrangementa Qfluld be mada with the atata tp 

QondviQt a large-aoale aomputet" veplflQatloni * 

' •■■ ' ■ , 

' Exhibit 3.3 9ummarl«Ba , the' treatment aaelgnmenta for aaoh of the four 

treatment groupa. ' . 

In a number of oasea to meet the needs of a {!|arttQular aohoQl dlstrloti tha' . 
aoalgnmont methoda outlined above vyore modified. To radiioa the burden In, 
low-oriroUment aohool dlatrloto, a emaller cjurnber of treotment groupi waa 
Implemented. J .. . 

As hoo beon notedi oohool diatrlcta thqt hod already comploted thalr application 
proqoaa when they a^greed to partlclpateMn the study preaented a special problemt - In 
these achool dlatrlota^ It was not poaalbla to Implement Quality Aaaurance 
Procedurea I (IVPP application), H (documentation with application), or III 
(documentation conalstency check). For thla group of acHool diatrlcta, 6 modified 
aet of treatmenta waa davaloped. I^choola were atatlstlcally matched anc| randomly 
assigned to one of three groupa* Th© groupa were atatlstlcally matched In terma of 
grade level and enrollment* " Referring to Exhlb^ 3.4, in Group CI the evaluation 
contractor supervised the drawing of a 3 percent random /sample of applications.^ 
School personnel then conducted a parent, telephone conference with sampled 
applicants to verify application Information (Quality Asaurance Procedure V). In 
Group C2, the evaluation contractor supervised, tbe drawing of a 3 percent random 
. sample of applicants. Recipients 'were required tb provide documentation of Income 
(Quality Assurance Procedure IV). In Group G3, 5 percent random sample was also 
drawn. Applicants in this group were required to either provide documentation of 
income (Quality Assurance Procedure IV) or to sign a release allowing third-party 
checks of their reported income (Quality Assurance Procedure VI). / 

In summary, an experimental design was used because of Its capacity^ to 
determine, the true effects of the pilot quality assur^ance procedures by controlling 
external factors that could either bias the results or provide alternative 
explanations for^ observed outcomes, the design featured: 
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EXHIBIT 3.4; SCHOOL BUILDING ASSI6NNENT TO TREATMENI GROUPS IN THE EXPERIHENTAL SfHOOL DISTRICTS 
THAT DID NOT USE THE IVPP APPLICATION 
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• Testing pll quality assurance procedures in each school district to 
eliminate the potential for bias due to the Interaction of school district 
characteristics and quality assurance proc , 

• Independent validation of WPP application Information and c|uality 

. assurance procedures through In-home audits; and ^ 

• Independent measurement of the possible deleterious effects of the 
quality assurance procedures on program participation by\ eligible 
households. 

Appendix B 'Summarizes the site-specific characteristics of the experimental school ^ 
districts. 



ENDNOTE - 

School districts that did rjot use the IVPP spplicBtiorrr^we not able to test 
the new application (Procedure I), require dQqump\)tBtlon with the 
application (Procedure 11), or perform the documenti^^^^ 
(Procedure in). In these school districts, schoolA, were divided Into three 
groups; one group conducted documentation after^ligibility determination 
(Procedure IV), one group performed parent telephone conferences 
(Procedure V)J and in the last group, sampled househplda, were given the 
option of providing documentation (Procedure IV) or of signing a waiver to 
permit third-p^ty verification of information . on. the application 
(Procedure VI), » 
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PHASE n DATA COLLECTION 



Phase 11 of ^VPP^ collected a wide variety of data for use In evaluating the 

^quality assurance procedures. Data were collected on the school district/ school, 

y ■ ■ . ■ ■ • .\ ■" ' - V "'^' 

and individual applicant levels. ^ , 

The primary data collection instruments were: 
School-District Levej;^ tt^,^ /• , 



• Experimental 8C?hool district questionnaire ^ \ ; ^ % 

• 'Control school district questionnaire 1 £$ 

• Experimental school district monitoring visit report ' , 

School-Building Level (In Experimental School Districts Only) 

• ' School summary report on application processing 

• School summary report on verification procedures 

Applicant Level (in Experimental School Districts Only)^ 

• ^ Application ' 

• Verification outcome report > , 
^ • In-lio me audit 

In addition, the evaluation contractor maintained records of communications with/ 
experimental school district personnel concerning problems encountered y/hile, 
implementing the procedures and records of steps necessary to implement 
State-level collateral contacts. ' >^ ^ ^ . 

This chapter describes the data collection instruments and methods that were 
used. ■ ^ > ' 
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M: EXPERIMENTAL SCHOOL pISTRrc^ 

The experimental scHdol district questionnaire ^as completed by 29 of the 
30i/ experimental schooL districts '^fl^^d returned by mail following the completion 
of all verification activities. The questionnaire obtained data on the following: 

• Participation in schooLmeal, programs by school in School Years 1982-83 
X and 1981-82; ®( 

* • Administrative procedure? used for processing applications and conducting 
, verifications; ^ V 

, Administration and cost impact of the quality assurance procedures; and 

. Reactions to the quality assurance procedures aihd suggestio9s for 
, improvement. 

4.2: CONTROL SCHOOL DISTRICT QUESTIONNAIRE 

• Tl^e contror school district questionnaire (Form C) was completed and returned 
V mail by 87 of the 902/ . control school districts. The questionnaire obtained 
data on the following: 

• > Participation in school meal programs in school years 1982-83 and 1981-82; 

• Administrative procedures used for processing applications; 

• Application verification procedures used (this section was included only in 
the nonexperimental school district control state questionnaire); 

• Administrative costs of application processing and verification; and 

• Opinions concerning alternative application verification procedures. 

4-3: EXPERIMENTAL SCHOOL DISTRICT MONITORING VISIT REPORTS 

Monitoring visits were conducted to all experimental school districts. The 
monitoring visits had three purposes: assure procedure integrity and consistent 
Implementation; dtStain anecdotal data on school staff experiences with the 
procedures; and assist school district and school building staff in implementing the 
procedures correctly. During the monitoring visits interviews were conducted with 
the school district directors, individuals who processed applications in schools, and 
interested school district administrators. During the interviews, the application 
processing and verification procedures were reviewed, and problenris encountered or 
anticipated were discussed. A sample of applications and supporting documentation 
was reviewed. 



V i4.4:'-:SCHOOL SUMMARY REPORT ON APPLICATION PROCESSING 



M ^ The school summary report on IVPP application processing was completed by 
^* scfiool staff in experimental school districts immediately following processing of the 



^4 
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application for school meal bepefits. Approximately 400 Forms were completed in 
i4 of the 25i/ experimental school districts that used the IVPP application. The 

• ; form-^obtained claJta on the time spent processing IVPP applications, the number of 
' ^applications processed, problems encountered, and expediences of schools requiring 

documentation with application. 

/ 4^5! SGH60L SUMMARYvREP0RT ON VERIFICATION PROCEDURES 

I The 3chool summary report on verification procedures was completed^twice by 
school' staff in experimental school districts. It was completed immediately 

Ijpillowijig first-stage verification activities (document consistency check, follow-up 

* '• * ■ ■ " ' ■ ' 

' ' dodgnpientalion request, or parent telephone conference). The form was completed a 

" '>• ^ ■ ■ ^ ' ■ • ■ 

/'second time following local third party contact verification. Approximately 460 
forms^ were; completed in the experimental school districts. The form 

V- db^i^ned data on ti procedures, the number of applications 

. improvements 



■verified, problems encountered, comments on the procedure, and suggestions for 



^4.6SIVPP yQ^ND>JpN.IVPP APPLICAT^ 

N : ^ ^Twenty- fburi/ of the 29 experimental school dikricts used the IVPP 
ap^licatiori.'-^The^ other five pilot sites used their own local application form. Every 



application that was verified using one of the quality assurance procedures was 
'ab8|ibacted; |^pproximately 11,600 IVPP application farms and 3,800 hon-IVPP 
applihatitjo'^orms were collected and abstracted 
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■Ml dll3ALITY ASSURANCE PROCEDURE OUTCOME REPORTS 

-,"/^' / , . • 

A\ verification outcome report was , completed by school district staff every 

^^nje a verification was conducted: 

i • Document consistency check (Quality Assurance Procedure III); or 

I' • Follow-up documentation request (Quality Assurance Procedure IV); 

^''^ • Parent telephone conference (Quality Assurance Procedure V); 

• Local third-parfy check (Quality Assurance Procedure VI). 
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A total of 539 document consistency checks, 2^695 follow-up documeritatlon 
requests, 2,885 parent telephone conferencel, pnb 3,833 third-party check outcome 
reports were completed/ The outcomev^ about th^, type of 

procedure performed, verification informatfen obtained, discrepancies discovered, 
actual time spent completing the verification, and effects on program eligibility. 

4.8 IN-HOME AUDIT ^ 

In-home audits were conducted to validate information on income and household 
size contained on the IVPP appUcations^and the non-IVPP applications used in the 
experimental/school districts. In-home audits were conducted in 152/, of the 29 
experimental school districts. They were not conducted in the control -school 
distfictsl The in-home audits provide the basis for determining the true error rates, 
These were .used to assess the deterrent effects of the IVPP application as' well, as 
the detection effects of the other quality assurance procedures. In addition, the 
in-home audit serves as the basis for validating the actual conduct of procedures by 
school officials as well as recipient reactions^ to and difficulties associated with the 
use of each of the quality assurance procedures, therefore the In-home audits had 
four functions— measuring errors associated with deterrence; measuring errors 
associated with detection; validation of the quality assurance procedures; and 
examining the respondent's assessment of feasibility of thp procedures. 

To improve the quality of the in-home audit data, respondents were asked to 
sign a statement certifying that they agreed to provide accurate information. All 
in-home audit respondents were advised that the information they provided would be 
used only for statistical purposes and would not.b^^^^^^ by school officials to change 
their eligibility. During the course of thejintem^^^ respondents were asked to 
supply documentary evidence (such as check stubs, program eligibility certificates, 
etc.) for every source of income identified during the audit. This documentation 
was used to validate the income they reported on the meal benefit application 
form. Household size was validated by requesting the respondent to enumerate all 
househ^d members. 

The in-home audit survey resulted In 2,093 completed in-home audits, which 
produced verification data for 3,767 meal benefit recipients. Of these, 1,217 were 
completed in school districts that used the IVPP ■ application. An additional 547 
audits were conducted in school districts that did not use the IVPP application. 



Of the audits attempted with program participahtSi only 3«25 percent resulted In a 
refusal to cooperate* ^' 

Using data from completed in-home audits, a model was developed In which 
sippllcation Information predicted which applicants underreported their income or 
family size to receive benefits in excess c^f their true eligibility. This model was 
then applied to application data from audit refusals to estimate the percentage §f 
refusals receiving excess benefits.^ Comparison of the estimated excess benefjit 
rates for, respondents and refusals was used' to estimate refusal rates for those 
receiving excess benefits and those not receiving excess benefits. (The 
mathematical details of the analysis ^^are. presented in Appendix C.) This anaysis 
found no significant difference for refusals among eligibles and ineligibles. 
Therefore nonresponse did not bias the estimates of error used in this report. 




... :.' : ■'ENID NOTES , . ' 

. "V ■ . ' • ■ ■ •• . 

1/ 'One of the 30 experimental school districts discontinued participation in the 
** study based on advica. from its legal counsel. 

2/ Three of the 90 control districts failed to complete the questionnaire, even 
~ though they agreed to do so when they were recruited to participate in the 
study. , ' 

3/ In-home audits were conducted in 15 of the 29 school districts instead of all 
" experimental districts to minimize data collection costs and logistical problems. 
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EFFECTS OF QUALITY ASSURANCE PROCEDURE I-THE IVPP APPLICATION 



Quality Assurance Procedure I, the ;IVPP application/ was designed deter- 
misreporting. The results of a test of this procedure are reported in .this chapter* 
' Similarly, Quality Assurance Procedure II, the IVPP ajpplication with documentation, 
.,w^^^ to teat the additional deterring effect of requiring applicant^: to 

p^^^^^ the IVPP application; ^hapter 6 presents the test 

; rQ3u|ts^^ procedure. In contrast. Quality rA38urance Procedures III through VII 

r^^?^^^ detect misreportirlg-error after the application was approved. The 

effects of these pniiGtedures are " 




Virtually all- Bchodl districts vxe^^ to. apply for frii^loF 

reduced-price schobl meals at the Jg^egiiinin^ school year. School staff who 

act as the determining officials (prinK^pilY^^ their staff) are 

accustomed to reviewing and processing appiic^^ time. 

, The Income Verification Pilot Project tested the effectiveness and feasibility of 
using a redesigned free and reduced-price meal application (hereafter i^Terred t as 
the IVPP application). The quality assurance objectives of the IVPP application 
were to reduce income .misreporting that results in excess benefits and to obtain 
detailed information for subsequent verification using any of the quality assurance 
procedures discussed in Chapter 2. The procedure was tested at 24 experimental 
school districts, in 316 schools, and Involved approximately 51,000 applications. The 
detailed characteristics of the IVPP application are discussed on page 9 of the 
report. A copy of the;appiication is shown in Appendix A. 
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^ ,;"^:Approxiniateiy 251,000 IVPP apjjUcations were distributed to students in the 2^, 
'.school.' districts that used the IVPP/application. Of these, 166,000 were received byv 
if ftiauseholds , that were only requirecil^ IVPP to complete the application. The 
^■ remaining applications were distributed to households that were also required to 
,v complete th0^ application an<t:*<Hdoj^ent adult income sources-Quality Assurance 
Procedures li^-the subject of 4hapter 6. This chapter presents findings about the 
' feasibility anci costs of using the IVPP applicat;ion, the effectiveness of the IVPP 
^ application in deterring misreporting, and the effect of the IVPP application on 
program particijjation. 

- . 5a THE FEASIBILITY OF USING THE IVPP 

Becaus^'fhe IVPP application was designed to deter misreporting and facilitate 
verification income statements, detailed information was requested of 
• applicants. 'School -officials who determined eligibility based upon information on 
--the application had to review this information for completeness. 

In addition to distributing applications, responding to applicant questions, qnd 
collecting tcbmpleted applications, school officials were confronted with new 
requirements on .the application. It required detailed income by source for each 
adult in the applying household, contained' a detailed and explicit release statement 
that would permit verification, and requested the signatures of ' all adults in the 
household- features 'designed to reduce misreporting. Applications used in prior 
years did not>contain these requirements. 

yyi ■ ... . . 

■v) ' ' " ■ ■. ■ ' ■ . 

5.1.1 Admi nistrative Time and Cost of Processing the IVPP Application 

■■■ ' — ~~ ' " ~ ' ~ 

. petermlning officials reported that school personnel spent an average of 16 

r^^^^^^ and approving each of the average 150 IVPP applications' 

submitted by households; this is an average time spent by school officials in ah 

; average school. Overall, the average time per school required to process a single 

application varied from a low of 6 minutes to a high of 22 minutes. 

The tyjjBs of school personnel involved in processing the IVPP application are 
■ shown in Exhibit 5.1. Of the total time.spent processing applications, 37 percent 
was expendejd by principals,^ 46: percent Jby secretaries, 10 percent by teachers, and 
only 1 percent each by administrators and directors of school food services^./ 



XHIBIT 5.1: PERCENTAGE OF TIME .SPENT BY SCHOOL AND SCHOOL DISTRICT PERSONNEL 
, . PROCESSING THt lVPP APPLIC^^^ . . / 
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Based on rep oris frdm 87 control districts, this pattern of staff involvement In 
processing the IVPP applications was generally repeated' in the school districts that 
did not use the IVPP application. On the average, 28 hours were spent distributing, 
reviewing, and approving meal applications in schools not using the IVPP 
application. The tirne required to process the IVPP application was 52 percent 
greater than the time required to process a non-IVPP application in comparable 
school districts 

School districts participating in the IVPP project (and more generally, in the ^ 
school meal progranrw) a[?8orb the cost of reviewing and approving the free and 
reduced*price school meal application. Currently, there is no direct federal subsidy, 
provided to school districts to offset these costs.!/ In sites using the IVPP 
application/the unit cost of processing, (i.e., distributing, reviewing, and determining 
eligibility) a single IVPP application was $4.53. In matched control sites that did r\ot 
use the IVPP application, the correspondiiig cost was $3.05. These estimates inclu^ 
the time and materials needed by school districts to distribute the application, 
review* it, and make the eligibility determination. School district funds were used to 
pay for these costd. SFAs that did not use the IVPP application also used school 
district funds. 

5.r.2 Processing Probiems and Reactions to the IVPP Application by - 
School Officials ~ '■ ~" 

, ^ 
In schools that used tha IVPP application, an average of 150 applications were 

received, of which 39, or 26 percent, were returned tjecause they lacked the Social 

Security numbers of b11 adults, income of all adults, or appropriate signature(s)jL/ 

However, school officials reported that an average of less than one application per 

school was denied because the application was incomplete (e.g., missing the 

signature of an adult member of the household, missing household income, or missing 

adult Social Security numbers or not returned within a reasonable time. 

• Exhibit S.Z-'displays determining officials* reports of the types of applicants and 
staff difficulties that occurred in sites using the IVPP application. Overall, 40 
percent of the 269 reporting officials that only processed the IVPP application had 
received not even one complaint about the application Jl/ Sixty percent reported 
receiving one or more complaints. Of these officials, 44 percent reported that at 
least one parent did not fully understand the appiication, and 16 percent reported 
that at least one parent did not wqnt to complete the application. 



BHIBIT 5.2: SCHOOL DISTRICTS' REPORTS OF PARENT ; AND STAFF DIFFiCULTIBS ENCOUNTE 
; ilHEN USING THE IVPP APPLIGAT^^^^^^^^^^^^ 



60 
54 
48 
42 

,36 
30 
24 
18 
12 
6 
0 



III 

! II 

?\'\ 
'ill' 



t • • « 



1*1 



• • • S 

* * * I 
till 



III 



I 11 I 



1 1 1 1 1 1 ! * 

Ii:i 



I'.t'i 



|| ii*i 
1 1 1' I S' 



i!i!i!i!i!!!i 



•!':i!i!t!i!i 



I : I : I 

!!! 
lit 



' ' ' • 

* ' I h 

• III 
• • • I 



4 I l{l 

'ill!;'! 

Ml 'i!i 
•••••• 

*!i 

>|i i|* 

I I * 1 1 

4 li^ll 
• • • • 

i!i i!i 



I *iS * I 



i!:::!i!i!i!> 



•;i|i:i:i:«:< 



.:!!! 



•!•!• 
Ill 
I iji 



I I i ' I* I 



• • • • 
iii'i 



• !il iS 



..ii 

k*|l| '|l 
* 1 1 1 I ' I 

ill Kl' 

i!i!t;i 



1 1 1 . 1 1 1 1 1 : 1 : 1 : 1 : 1 ; I . 

'• ' I ' • 

III 

' ' 
i|i 



i;.|. 

I i!i 



• ;i:i 



i;!;! 

! : : 

I I ! I 



M' *i' 
iiM^i 



••ii^!;:!:i!;' 



II,. 
Ill 



I ii: i:i 
I ' I ' I ' I 
•••I 'I 
S' • • 

*|| • I 



•!'!•!•! 
!:•!•!•! 

'•'ill 

iJiiiii! 



*ii S* 



i;i I. 



1 . 1 . 1 1 1 A I : I : I : • : < 1 1 1 1 



iSlili I 
'iiiiii'i 

I i:i!«Si!i|i! 
iM 



'i"|i|iii|"i'i 

I i!i I i!i!i! 

I ill II l! 

I I II II ij 
i.i|i I i!i i! 
I 'i ' I ' i' I* »' I ■ 
•Ji,'ili|>i» • 

'i' 'I'W' • 

iiifii|i ip * 
i!5 I iWi • 
iji I I 1*1 I 

'i!i!i!i i!i I 
I I I I I I I 

: I ! 1 1 1 ! 1 1 1 ! 1 1 * 




! ! I 
iMi 



*!||*|||||| 
I I iJi I I 

i;i 1^1 I I 

ill •»ii*ii 
I I ill 11 

I I 1^1 I I 
i:i!i;i;i!i 
1*1 '1*1 'i ' I 
|| iiSii I 

I • Ml' 



I I I I 11 
I II II * 

'|i 'l! 

I I I i 1^ 
i{i I I » I 

iiSiStiiii. 

I I i!i ii'i 
I I I I I I 
I I i|i I I 

I I iji I I 

I I §{1 I I 

I I i!i I I 

I I I I I I 

I I I I I I 

I I ijiji I 

I I I I I I 

I I I I I I 
i!i;i!i!i!i 

ill I I I 



•|!|'! 
!•:•! 



■ I ■ I ■ 
III 
III 
III 
III 
ijl I 

Mil 

ill I 



III 

i!i!i 



ifi I 
!•:!' 



ii|iti;iiiiiiiii 



l';!i!i!i!;!.!i!ii:ri!l 



! ! 

'III 

'!|| 

II 
||| 
lit 



i;i;i:ii 



. . . •!i|'!*!* 

Ial| lal|l|l!!i*i' 
ifriji I'lllllll 



, - . • • • • • 

I ll*lll*ll I ll 



• I I iji 

I I • i!i 

I I I i|i 

i';';';'l'!'!'l'! 

Si I I I I II' 



• ■ « • 
'i' *!ii 

I I 11* 



I'l'l'l 

,11,11:11:1:1: 



1 



2 



3 



4 



Type of Jflffifulties 



Determining officials reported that: 



1 - parents did not understand the application ' . 

2 - parents did not want to complete the new application 
3'- staff were unable to perforni other functions , 

I - staff dissatjsf action 
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Twenty-3lx percent-of the dBtermining officials Teported that Increased staff time 
requirements associated with the IVPP application prevented school staff from 
attending to other duties on a timely basis. Eleven percent reported staff 
dissatiafaction with some aspect of. application processing. It should be noted that 
complalnts-are-also-routlnely-recelved_about nonTiyPP.1^ 

are provided by school officials who were processing the more complex IVPP 
application for the first time. If the application were. used nationwide, the same- 
reaction^ could be expected ^ during the first year of use. However, they should 
diminish in subsequent years. . 

Overall, the IVPP application requires approximately 52 percent more time'to 
process \and approve than an application not explicitly designed to deter 
misreporting. However, the costs associated with the IVPP application could be 
expected to diminish over time. It should be noted that school district officials 
(instead of school staff who process applications) provided time and cost data in 
school districts that did not use the IVPP application. Therefore they may have 
underestimated time spent in the schools. It. is also evident that the IVPP 
applicatiqn caused some adverse reactions among school staff who processed the 
applications. However, t^e degree to Which these reactions differ from reactions to 
non-IVPB'applicatiqns is not known. 

• ' >'••""■. . ■ . ■ 

5.1.3 Applicant Reactions to the IVPP Application 

A sample , of program participants who completed the IVPP. application v/as 
surveyed. .The IVPP application represented a dramatic change from previous years 
in the amount and detail of information that was required of the free and 
reduced-price school meal applicant. New requirements included a detailed listing 
of all household members and their 'respe6tive incomes for as many as six adults; 
indication if the household was receiving food stamps; and a release authorizing, 
verification by school officials. In the prior year, most applications asked only for 
total hckjsehold income, 'family size, and the names and Social Security numbers of 
all adults and the signature of the adult completing the application. 

Households that applied for Jfree or reduced-price school meals were asl<ed to 
recall the features of the application that distinguished it from prior applications. 
Overall, most applicant households cleatly recalled most of the neNy features of the 
application. Over 70 percent noticed each of the following features: the 
requirement for Social Security nuhibers; the requirement for detailed and current 



monthly income reporting} the requirement for food stamp benefits reporting; and 
the requirement for verification authorization. Thia pattern was true of households 
that wera eligible for benefits (as independently determined by the in-home audit) as 
well as those that were ineligible. ' 

Yet when asked if any of these features of the IVPP .application caused concern . 
among the applicants, the large majority of applicants were not greatly concerned • 

' about the new application. Specifically, 87 percent were not concerned about being <| 
asl<ed to report adult Sociaj^Security numbers; 86 percent were not concerned about 
reporting current monthly Incomej 86 percent were not concerned about reporting 

' detailed income for each adult household member; 89 percent were not concerned 
about , signing a release authorizing school officials to verify information on the 
application; and 87 percent were not concerned about reporting the receipt of food 
iBtamps. 

The IVPP application also apparently presented some difficulties to the 
applicants who were able to recall specific details of the IVPP application. Thirty 

• percent reported that the IVPP 'application took more time to cortiplete than 
previous applications; 23 percent found the application more difficult to complete, 
while 21 percent found the application more confusing; and 17.5 percent thought 
they were more accurate when completing the application. However, only 19 
percent of the applicants were less sure that they would qualify for free or 
reduced-price school meals. The same general pattern was exhibited among 
applicants who were eligible to receive free or reduced-price school meals and those 
applicants who were ineligible. However, households that received excess benefits 
were less sure they would receive benefits after completing the IVPP application 

'^^tharTvv^ truly eligible households. Eighteen percent of truly eligible, households 
were in doubt of their eligibility compared to 33 percent of the misreporting 
households that were in doubt. While not definitive, this finding does suggest that 
the application may deter misreporting among' households with Incomes exceeding 
eligibility limits. \ \ ' 

In summary, both determining officials and applicants reported some difficulty 
associated with the. IVPP applicat4erv_School officials tended to suggest that 
applicants had more, problems than the applicants themselves reported. It is likely 
that some school officials may have Inflated the degree of difficulty because they 
were opposed to the more complex IVPP application. In contrast, applicants may 
have understated their difficulties because they were asked to recall an application 

erIc 5S 



they oubmltted aeveral months ago. Therefore It appears reasonable to conclude 
that the application caused some difficulty, but that the extent of difficulty was not 
precisely measured. Further, It should be noted that these reports of problems are 
associated with the first experience with a new application. These problems should 
diminish in subsequent years as both school officials and parents become more 
accustomed to the reporting requirements. More Importantly, the IVPP application 
had a greater Impact on. applicants who mlsreported Income to obtain meal . 
benefits. As shown lat^r In this chapter, the IVPP application significantly reduced 
error rates, suggesting that the IVPP application does deter much mlsreporting. 

5.2 EFFECTS OF THE IVPP APPLICATIpN ON PROGRAM ERROR RATES 

The IVPP application, designed to deter mlsreporting. Is a quality assurance 
procedure in its own rlght.6/ The effect of the IVPP application on overall 
program error rates cannot be evaluated directly because of the lack of a 
scientifically-selected comparison grou^ with a known error rate. Therefore the* 
best available comparisons for purposes of evaluating the effect of the IVPP 
application on error are past studies and Phase II sites that did not use the IVPP 
application.!/ ^ 

Exhibit 5.3 presents the error rates found in the ^school meal program based oh. . 
five independent samples from three previous studies and Phases! and II of the IVPP/:.^ , 
study. The verifications for both Phase I and II of the WPP study were conducted.^: • 
through in-home audits as described in Chapter 4. The verification in the National 
Evaluation of School Nutrition Programs was based on personal interviews that-wefe 
not supported with documentation. In thp case of US.DA's Office of the '^Inspector . ' 
General, study verifications were conducted by third-party wage checks and visits^to >j^^ 
employers. ^ * 

The first three rows of percentages in Exhibit 5.3 represent ;^apip^ants vijhp^e 
program eligibility was verified as correct. The fourth ;^nd fifth rows of Exhilalt 5.3^^ , 
(those receiving reduced-price benefits but eligible for fre^ meal benefits) ctjnsist op 
households who overreported their income or underreported their family sizej^^they \ 
did not receive the full benefits to which they were entitled. The next fiye^rovvs (6 ^ 
through 10) of the exhibit represent the groups of primary^ interest to ^thig, _c^j;r 
study— those whose mlsreporting of income or family size meant they receiv«d 
benefits exceeding those to which they were legally entitled. The final row<ll) of 
the exhibit shows the results of the five sample-based surveys of misreppryng^n 
school meal applications. 
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The first column of figures In Exhibit '5.3 shows the results for sites that used 
the IVPP application. As can be seen^ 11.7 percent of applicants were found by the 
In-home audits to be receiving excess program benefits. Row U reveala that this is 
by far the lowest error rate found In the five samples} the four other samples not 
using the IVPP application had Verified error rates of 50 to 25,0 percent higher than 
the 'error rate associated with the^HVPR appllcatlon.8/ Particularly - noteworthy 
are the 23.7. percent error rate found In the Phase II sites that did not use the IVPP 
application and the 17 .4 percent error rate found In the Phase I sites. The Phase I 
sites used a revl§ed application that^ Incorporated some features ^f the IVPP 
application (l.e.| more detailed Income Information and a warning of possible 
verification)} but to a lesq^er degree. Although the 23.7 percent error rate for Phase 
II sites that did not use' IVPP Is based on a limited sample, it represents an error rate 
associated .with the typical 1982-83 school meal application and strongly suggests 
that the IVPP revisions reduce the error rate substantially. This Is corroborated by 
the lower Phase I error rate (17.4 percent) which Is also associated with a revised 
application. ' l" '■ ' 

The total row for ineilgibles receiving benefits also reveals an important effect 
of the IVPP application. The verlf/ed error rate for this category associated with 
the IVPP application is 3 pei^centi whereas comparable verified error rates are. oyer 
300 percent higher.- This is important because the cost' savings associated wjth^ 
determining ineligibility is greater than the cost savings of correcting' eligibility 
from free to reduced price. In effect/4he IVPP application is best at preventing the 
njost costly types of mlsreporting. * T 

* These comparisons constitute strong evidence that the IVPP application has, 
; successfully reduced misrepor ting. "Because the Phase II design did not Include a 
statistically matched control group with a known error ratei no definitive estimate 
of the magnitude df error reduction attrlbuted.to the IVPP application was possible. 
However, it is evident that the IVPP application does reduce applicant error 
sulDStantially. ^; 

5.3 EFFECTS OF IVPP APPUCATION ON PROGRAM PARTICIPATION 

In SFAs using the IVPP application, participation in the free and reduced-price 
meal program increased by an average of 2.9 percent. Of the. SFAs using the IVPP 

application, 47 percent indicated an increase in participation and 53 percent a 

■. . ■ ■'•■•.»■'■■ 

decrease.9/ 



The effects of ,tho IVPFf applloatlon on paPtloIp|tlqn oan be partially evaluated 
by qomparlBona with partlolpatlqn In SFAa that did ft^^^^^ tha^IVPP applloatlon. In 
all but one of the atatUtlcally matched altea that did not uao the IVPP application, 
program partiolpatlon Inoreaaeda The average Increase In participation In thoae 
sites was 8.7 pBrcent. Gomparlaon of this number with the smeller increase of 2,9 
percent In SFAa using the IVPP application auggeata that the application reaulted In 
a 5.8 percent lower participation rate than would have otherwiae occurredf In view , 
of the IVPP application's effectiveness In reducing the participation of ineligibles, it 
is to be expected that participation would be lower in SFAa using the IVPP 
application than in SF As not using the application. It should be noted that the lower 
rate of participation was based on reports from a small numberlO/ of SFAa and 
therefore is subject to substantial sampling error. The study produced no evidence 
that the lyPP application adversely affected the participation of eligible individuals. 
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END NOTES 



y The remaining time waa flpent a variety of Indlvlduala, Ingludlng eahool foad 
^, oervloe ataffy oafetBrja workers, and aldeoi / 

2/ BeoausQ the Sohool Food Authority (SFA) Instead of schools In sites not using 
the IVPP applloatlon provided data about time to process appl|oatlonS| it la 
possible that sohool4evel Involvement m^^ have been undereatlmated and 
"". SFA-lovel. Involvement overaBtlmated, Also please note that the 52 percent 
increase in time Is based on group averages and not Individual school fig^^ 

3/ State agencies that administer the progr^ are allowed to spend administrative 
funds for verification if done at the State level. 

4/ It was possible to Identify adults 0 the age of 

each household member was requested. V ' 

5/ Reports used to develop these estimates were completed by one official (usually 
a school principal) who summarized the efforts and experiences of school staff. 

6/ Because the IVPP application was uniformly implemented in all schools In SFAs 
T that used the IVPP application, the Phase II design contained; no wlthln-SFA 
comparison groups that might ba used to evaluate directly the effects of the 
IVPP application on applicant misreportlng. The ability of ^the evaluation to 
analyse effects of the IVPP application within SFAs was sacrificed to achieve a 
uniform application that provided a basis from which the effects of the other 
quality assurance procedures could be measured, ^ " 

7/ Three of the five Phase 11 sites that did not use the IVPP application also had 
. in-hgma audits. Therefore error rates are based on. these three sites, which had 
a combined enrollment in excess of 160,000. 

fl/ A comparison of the error rates in experimental sites that did not use the IVPP 
application and the error rate in Phase I IVPP sites should not be interpreted as 
suggesting that program-wide error rates went up between the 1^81-82 and 
1982-83 school years. The Phase I IVPP : sites used simple error deterring 
procedures not used by the Phase II non-IVPP application sites. Also, the error 
rate for the Phase I sites must be interpreted 'carefully because of the limited 
sample upon which it is based. 

9/ These estimates include only schools that used the IVPP application (Quality 
Assurance Procedure I). The estimates do not Include schools assigned to- 
> Quality Assurance Procedure n since documentation could also act as a barrier. 
Changes in school meal participation are calculated by statistically contrasting 
participation in the year prior to the study with participation in the study year. 
The same month in e*§ch year is used. 

10/ The analysis was based on the 15 SFAs that met the following conditions: used 
the IVPP application/ reported participation rate data, and had a matched 
control SFA that also. reported participation rate data. 



EFFECTS OF QUALITY ASSURANCE PROCEDURE II-INCOME 
DOCUMENTATION WITH THE IVPP APPLICATION 



A second method to deter mlsreportlng of Income Information la to require 
applicants for the free and reduced*prlce meals programs to supply documentation 
proving household Irjcome when the meal beneflt appllcatlon Is submitted. Income 
documentation submitted with the IVPP application was required of approximately 
27,700 applicants In 142 achoola within 17 of the experimental school districts. 

This chapter describes the test of this procedure In schools that used the IVPP 
application. The procedure— documentation with application— is Intended to 
Increase Incremantally the deterrence effect of tli.e I^PP application. The 
procedure Is described on page 11. As the previous chapter demonstrates, when used 
alone the IVPP appliqatlon substantially reduces error but increases school district 
coats associated with distributing, ^receiving, and approving applications. The 
information derived from these documents was used to detect error(s) later in the 
school year as part of the document consistency check (Quality Assurance Procedure 
Mil), and the effectiveness of that procedure is discussed In the next chapter. 

The evaluation of Quality Assurance Procedure II provides Important findings 
, about the feasibility of Implementing the procedure and its cost to school districts, 
the ability of this procedure to improve the error deterring ability of the IVPP 
application, and the effects of the procedurjp on school meal participation. 



a FEASiBIUITY PF REQUIRING flQGUM WITH THE WPP 

Tlilfl Bftotlon flddpeawq queatlona rolflting to the feaBlblllt^f^of requiring free and 
reduoed-prloo achool moal eppUoantfl to Mppiy dooumantptlon with the application 
vepltylng Income. 'Unllka Inteplm regulatory requirements now In affabt, , thia 
roqulroment waa mandatory In the IVPP project. Failure to provide documentation 
wao doflnod ao nonoompllanoe with appUoatlon requlromonta, and the fiohool dlatrlot' 
waa prevented from determining ejlglblllty. Determining of flolala and other achool 
ataf^ found it necooaary to foilowup with appUoanta to onaure that documentation 
had been aubmltted and that It waa thp correct document. The faaalblllty of Quality 
Aaaurance Procedure IMa dlaouaaed below, 

6.1.1 AdmlnlatratlvB Time and Coat Aaaoclated with Documentation with 
IVPP Application ~ [ ,.■ 

One-third of the achoola in each aite where the IVPP application waa teated 
required that all appllcanta submit documentation with the application, that 
.aubatantlated information provided on the application. Because an experimental 
deaign- waa utilized, it waa possible . to identify the Incremental time and coat 
increaaea aasociated with the verification procedurea. 

: The added requirement of the document with the application increaaed the time 
required to determine eligibility by an average of 6 minutea per applicant (22 veraua 
16 minutea). Thia represents a 37 percent increase in time. This increase varied 
:frorn a low of 1.4 minutes to 21 minutes In somewhat comparable school districts. 
Thus In a typical school where this quality aasurapce procedure was In effect, an 
average of 61 hours was required to review and approve an average of 140 school 
meal applications. The average number of applications Received, however, Is 6-7 
percent lower than the number of applications received in schools that implemented^ 
Quality Assurance Procedure I. 

As noted in the^ previous chapter, principals and secretarial staff were 
responsible for reviewing M the IVPP applications. As shown in Exhibit 

/6.1, this pattern was constant in schools requiring income documentation with the 
application, noting the following differences: there was a 6 percent increase in 
principals' time; a nine percent decrease in secretarial time; a 4 percent increase in 
SFA staff time; and a 7 percent Increase in SFA directors' time relative to schools 
that required only the IVPP^application. ,^ 
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Ths rBBBon for Wiese dlffsrenoBS wbb that th§ cloQumentatlon with ippllGSfclon 
Inoreaaed the naad for mors asnlop sohool piPionneli PrlnQlpali often found It 
neoaaiapy to Qontiot parents to obtain th© doGun^ent, and they had tQ;ln§p§Qt 
oarefuUy ^aoh. dooMment to onoura that U eompilad with guldellnafl. SFA staff also 
found It neoeeaary to Inoraaaa thelp laval of aotlvlty In the applloatlon pavlew and 
approval prooaw to anoura that applloanta ppovldad the required doqumenti 

The unit oont of reviewing and approving the doQumentatlon with applloatlon 
wQu groBtor than reviewing and approving the IVPP applloatlon alone, Jhe Inoraaaad 
unit coot waa $1.57, for an average total of $6.10 per applloatlon with 
documentation. Thla ooat may be aomewhat exaggarated beoauBB It |e known that 
the determining offlolalo (primarily prlnolpala) are Inclined to uoo the document to 
aoalot In detel-mlnatlon of eligibility rather than olmply to make aure the document 
had bean aubmltted. Subatltutlng Income amounto and redetermining total Income 
and eligibility, which Is to bo done on the baola of a aample and only oa part of 
Quality Aaaurance Procedure III, would require additional time. The degree to which 
this Incorrect use of the document Increases' the time required to process an 
application with documentation Is not definitively known, but It Is not bellevod to bo 
a significant Increase. 

6.1.2 Administrative Problems Associated with Processing the 

Documentation with Application '.. ~; . - 

When reviewing and approving the application with accompanying' 
documentation, an average of 65 applications received (or 46 percent of the total) 
were returned to the applicants because the accompanying documentation waa 
incorrect or missing. In spite of this apparently high number, applicants persiatod in 
securing documents and an average of only two applications per school (1.4 percent) 
were eventually denied benefits because the applicant failed to provide the required 
documents An average of less than one application per school was denied because 
the applicaht was unable, to provide the required /Jocument witti the application. 
The result reinforces the finding that^more time is required to distribute, review, 
and approve the IVPP application when it must be accompanied by documentation, 
bn the othef hand, findings suggest that among applicants who do submit 'an 
application, virtually all ;>(97.6 percent) can provide documentation with the 
application. 
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An Imporfcant §l§mint iffsefeing fehi fiislblllty of pfqulrlng eubmlsslon of 
dflmimentafclon with the IVPP ipplleatlqn wai thi rtaQtlon of §Qheol offlelili who 
reviewed and approved meal benefit appllsatlona. Seventy<elqht paraent of all 
determining of flQlals who profleaaed the IVPP applloatlon ^raportad that they 
reoelved at leaat one Qsmplilnt from parents about the dogumentatlon requirement! 
However, when afll<ed to Identify dlffloultles enoountered while proQewIng the IVPP 
applloatlon Wl|:h dDoumentatlpno-that Is, obtaining a oomplete appllaatlnn with 
doQumentatlon and reviewing It to determine oompllahee with the doeumentatlan 
requlfdment and «^l|glbUUy b«fl©d on the eppllaatlan-'^aniy p©rot?nt of tha 
reporting offlQleU Indloatftd that appUainta did not undaratand the appUoatlon and 
dOQumentatlon roqulramonto and 12 paroent Indloatad that parents ware raluatant to 
complota the application and aupply dooumantatlon. With ragardsto'achool otaff^ 17 
porcent of the officials raportad that their staff ware unable to perform, routine 
dutlea bacQuoo of the extra time required to review applications. Nine percent of 
the officials felt that th^ application created diooatlafactlon among achoohataffy 
resulting from Incraaoed contact with parentBt 

■ ■ ' ' " ■■ • -. . A V Y ■ ' ■' . ; / ^ ' ■ ■ ■ / ■ ■ • ' 

6.1 A Applicant Reactions to Documentation Requirement with the IVPP . 
AppUcatldrT " "^'^ [ , ■ ' ^ ' ^ " 

In*»home audits were conducted with householda subjected to the documf^ntation 
with application quality assurance prcpcedure/ Eight y-aeven percent could not . recall 
any specific difficulty in obtaining the document required for the IVPP application. 
Six and one-half percent of these househgl^s reported that they had to contact a 
third-party agency (e.g^ welfare office, employer) to obtain, a copy of the 
document. ^ Among these same householdSi the documentation with application, 
requirement did not raise a substantiaLnumber of concerns. Eighty-three percent of 
these applicants reported that although they recalled the requirement^ it did not ^ 
raise any significant concern or problem for them. 

6.1.5 Types' of Documents Submitted with the IVPP Application . 

The. types of documents submitted with the IVPP application are summarized in 
Exhibit 6.2. Wage documents (such as check stubs and pay envelopes) accounted for 
45 percent of the total. D.ocumentatlon of varioua unearned Income sources 
accounted for 41 percent of theAjpcuments and included these types of benefits; 
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EXHIBIT 6.2! TVPES OF DOCUHENTATION WITH APPLICATIflH SUBHIITEO it APPUCAHTS FOR FREE AND REDUCED 
PRICE SCHOOL HEALS 
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Aid to Families with Dependent Children (17 percent), Food Stamps (13 percent), 
Unemployment Compensation (7 percent), and Supplemental Security Income (6 
percent). Five percent of the applicants provided a written explanation of why they 
had reported zero income on the application. 

These findings suggest that applicants are able to obtain and provide 
documentation with the application. More importantly, wage documents constitute 
a major portion of these documents. This enables school officials to verify the most^ 
errorrpronelsource of household income. , ' 

6.2 EFFECTS OF DOCUMENTATION WITH IVPP APPLICATION REQUIREMENT ' 

ON PROGRAM ERROR RATES 

In 14 of the 20 school districts that implemented this procedure^ applicants 
were sampled for in-home audits. A matched 'sample of audits of applicants not 
required to submit documentation wijjh the IVPP application was also conducted. ' 
The documentation requirement apparently had a relatively . small effect on 
misreporting. ' * 

Exhibit 6.3 displays the results of the Audits, .which identify error rates. This 
exhibit shows that applicants required to submit documentation with the IVPP 
application had a lower overall excess benefit rate than applicants not subjected to 
the requirtsment (9.4 percent compared to 11.7 percent, respectively). The observed 
difference in exqess benefit rate)is only marginally significant statistically at the .1 
level. A comparison of error rates for ineligible households who received benefits ' 
reveals that the* IVPP application with documentation resulted in an error rate ^ 
approximately half of the error rate assdoiated with the IVPP application without 
documentation. The application with documentation does deter nriisreporting more 
than the application without documentation. However, the increrhejital effect of 
documentation in reducing errors is small relative to the effect of t^a application 
"'alone. . ^ - ' ^z- ' ''''' I ' 

6.3 EFFECTS OF DOCUMENTATION^W^iw 

^:r - PARTICIPATIOrsJ . ^ ^ ...v; -f ^ -'I; y|> ■ ; , 



f In all districts that included schpofs. requjrth^dd^^ yyith the/T 

application, schdoj^eal program iparticijDation was loj/^ar ^f^^'r^ 
than in "nondocumentation'* schools. On*tHe averageyschools fequirlfig ciocu 
with the IVPP^application^experienced^ri 8.7 pelr^nt lower participjatidii ir^ 



EXHIBIT 6.3s EtlGIBILliy STATUS BASED ON APPLICATION DATA JO «f"^^^ "^^'"^^ 
00CU«NTAT10(i WITH THE IVPP APPUWTION COfffAREO TO SCHOOLS TW^^ 
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V Heqativ'es indicate that the IVPP application with docuineniation has lower error percentages than the IVPP application 
without documentation. , ' 



. * , ■ % . ' ■// • " ' 

schools not requiring it (and 14*5 perqft'nt lower than schools not , using the IVPP 

■ ■ • • ^ " ■ ■ ' ' ■ ' ' ' 

application). The overall effect of the documentation requirement on participation 

ranged in extremes from only 2.2 percent to as much^as 17.Q percent. 

■ ■ \ ■ \ . ■ ; ■ ■ . 

The effect of the documentation requirement on program participation can be 
.seen as the sum of its deterrence and barrier effects. Deterrence occurs when the 
documentation/^Tequirement dissuades ineligible households ^5 from applying or 
misreporting eligibility status. A barrier occurs when othery^^^eligible households do 

not apply for program benefits because of the documentation requirement. 

V . ■> 

There is strong evidence that the dociimentation requirement resulted in both 
barrier and deterrence effects. As noted above, in schools requiring^ocumentation 
with application, 1.7 percent of program participants were found to be ineligible by 
the in-home audit. This contrasts with a somewhat higher ineligibility rate of 3 
percent in schools not requiring documentation^ This? reduction, in the numBer of 
ineligible program participants cannot account for most'of the 8J percent reduction in 
program participation associated with the documentation requirement. Even if the 
documentation requirement had eliminated all ineligibles from the program, this could 
account for only a 3 percent reduction in program participation. Thus deterrence is 
estimated to account for only 1.4 percent of the 8.7 percent reduction in participation 
associated with the documentation requirement. The remaining 7.3 percent ia due to 
barrier effects. This means that for every ineligible household deterred from 
misreporting because of the documentation requirement; more than five eligible 
households did not participate in the program. 
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EFFECTS OF QUALITY ASSURANCE PROCEDURES III THROUGH VII- 
IMPLEMENTATION AFTER APPLICANTS APPLY FOR SCHOOL MEAL BENEFITS 

In contrast to the IVPP Application and Documentation with Application 
procedures just reviewed, the IVPP tested five quality assurance procedures 
designed to detect income mlsreporting after applications to receive free and 
reduced-price school meals had been approved. Each of the five procedures differs 
on the basis of verified information. The*document consistency check (Quality 
Assurance Procedure III) and follow-up documentation (Quality; Assurance Procedure 
IV) focus on a single document relating to the major source of household income. 
Parent telephone conference (Quality Assurance Proc^ure V) is a method of 
verifying all forms of income as well as household size— the determinants of 
eligibility. Third-party checks at the local level (Quality Assurance Procedure VI) 
tend to be restricted to verification of eligibility for welfare, whereas State 
third-party checks (Quality Assurance Procedure VII) can include wage and welfare 
benefit verification. ^ 

Phase II of the Income Verification Pilot Project tested the feasibility and 
effectiveness / of each of these five quality assurance procedures. The 
characteristics of these procedures are described earlier on pages 12 through 17 of 
this report. Quality Assurance Procedure III was conducted in 70 schools and 
involved 539 recipients. .Quality Assyrance Procedure IV was conducted in *^234 
schools and involved 2,695 recipients. Quality Assurance Procedure " V was 
conducted ' in 315 schools and involved 2,885 recipients. Quality Assurance 
"Procedure VI was conducted for 3,833 recipients, while Quality Assurance Procedure 

VII was conducted for 1,933 recipients. The remainder of this chapter presents 

- * .■ * ■ ■ 

findings on the implementation and administrative feasibility of the procedures, the 
ability of these procedures to detect misreporting that results in excess l^pefits, 
the cost of conducting these procedures, and their impact on free and Ted^j^^irptice 
school meal program participation. / > 

O 
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ViAMPbEMENlTATION Ahldi: FEASI^l-ITY OF QUALITY ASSURANCE ' 

• ^pRdcEDUREs, ni^THFigucaWr ' ^ ' : 

'■, "JilVj^^ conducting verification for a_, sample of 

/^•s^'^^wved-'^rei^lpleh^^ the procedures were actually 

j^/'iraptejnnente,^^^ reafc^tions of' recipients. Cost and time findings are treated in 
^;Sectlon7,3^''^' .■ '. i'm '• " ■ 

Problems Assoclateg with Procedure Implementation 

In Phase II bf ' the pilot, project, the evaluation contractor provided substantial 
: training and technical assistance" to each experimental site.. Manuals were 
developed with step-by-step instructions for conducting the quality assurance 
procedures. Sessions were held in all pilot sites to train the school district personnel 
who conducted the procedures. Each training session averaged four hours. 
Follow-up monitoring visits were conducted, and "toll-free telephone technical 
assistance was made available. The .quality of imprementation was monitored 
throughout the demonstration. Reports were completed and filed on application 
processing, procedure activities, outcomes, and program participation. Finally, the 
in-home Audits provided an independent cross-check of the integrity of . verification 
procedure implementation. That is, applicants were asked to confirm and. describe 
the (Procedure they were selected to undergo. Despite these efforts, the verification 
procedures were hot fully conducted as specified. 

The moat extensive problem occurred with the document consistency check 
procedure (Quality, Assurance Procedure III). The experiment was designed to 
separate sharply the procedure of requiring income documentation with the 
application for all applicants frprti the procedure of checking information 
consistency on the document and application for a sample of applicants following 
approval for meal benefits. These two activities were separated to permit distinct 
analyses of how well the documentation requirement deterred misreporting as 
opposed to how useful the ddbumentation was itself in detecting misreporting. 
Unfortunately, there is strong evidence that officials in many schools used the 
documentation supplied with the application to change or correct reported income. 
The distinction between the documentation with application procedure (Quality 
' Assurance Procedure II) and the document consiste/icy check procedure (Quality 
.Assurance Procedure III) was blurred. The result of this implementation problem is 
that tha findings will tend to overestimate the efficiency of the documentation with 
application and to underestimate the efficiency of the document-consistency-check 
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procedure. The implementation problem does not explain the high barrier effect 
associated with Quality Assurance Procedure IlVhowever. 

The parent telephone conference also experienced implementation problems. 
There is evidence suggesting that school officials had difficulty verifying income on 
the IVPP application. This difficulty manifested itself In various ways. First, some 
school officials confirmed Income reported on the IVPP application rather than 
asking the recipients to state their income. Second, Instead of verifying, the 
household's Income when they applied, school officials asked for the household's 
current Income. Each of these problems undermined the procedure's effectiveness. 
They also influence the findings relative to the time and cost associated with the 
procedure. Time and cost estimates presented later are probably less than what 
would be found If the procedure had been Implemented faithfully.* 

1 The second more^ critical implementation issue Is that many of the prescribed 
parent telephone, conferences (Quality Assurance Procedure V) and follow-up 
documentation requests^ (Quality Assurance Procedure IV) were never conducted. 
Forty-two percent of households selected for parent telephone conferences reported 
they were never contacted by school personnel, as revealed In the In-home audit. 
Thirty-one percent of those households selected for follow-up documentation 
requests reported they never received , a request ' for income documentation. 
Undoubtedly some households who reported having never been contacted may have 
simply forgotten because the audit w^s conducted several' months after the 
application period. Qthets may not"^ have wanted to admit the schools contacted 
them. However, there Was a very strong clustering of the reports of noncontact by 
^ school district and school. For example. In one school district, 71 percent of those 
selected for parent conference, verification reported not having received a call from 
schobl/; district personnel. In* another ^chool district, the proportion not reporting 
contadt fell to 3 percent, ' ' 

Failure of the school districts to conduct faithfully many of the prescribed 
procedures In spite of extensive ^ training *and monitoring raises an Important 
' methodological and substantive Issue, The methodological Issue Is how to measure 
the outcome of experimental treatments that were only partially Implemented. The 
decision was made to measure verification effects using only those cases for which 
there were verification outcome reports. TJjjus the findings are measured, results of 
verifications actually conducted and not results of verifications prescribed. 



Failure of school districts to conduct meriy of the prescribed verifications is ah 
important study finding. Despite voluntary cooperation in the study by school 
districtSi extensive training^ techQical assistancei and monitoringy jiigh levels of 
noncompliance were found. Therefore it is reasonable to expect that federal 
regulatory requirements of meal benefit application verification are likely to 
experience as high* or higher noncompliance rates if not accompanied by even more 
extensive enforcement, training, technical assistance, and monitoring activities. 
This finding must be viewed as suggestive rather than conclusive since r^^federal 

requirements were in place to test compliance. 

... - . . ^ » 

7.1.2 Program f^articipant Reactions to Quality Assurance Procedures 

Audits of participating households in the sites that tested the procedures 
included a series of questions designed to measure reactions to the following income* 
verification procedures: document consistency check, follow-up documentation 
request, and parent telephone conference J:/ As shown in^S^hlbit 7.1, the findings 
indicate that the majority of free and reduced-price school meal recipients were not 
significantly concerned about having their applications verified. Specifically, among 
verified households, 78.0 percent did not express a significant concern about 
document consistency checks; 67.4 percent expressed no concern about the parent 
telephone conference; and 63.3 percent expressed no concern about the follow-up 
documentation request. Expressed concerns such as ''verification was an invasion of 
privacy'' or ''confusion as to why the application requested the information" were 
cited l^y less than 10 percent of the verified households. 

The in-home audits of free and reduced-price meal recipients suggest that 
providing documents (either with the application or later in the school year) did not 
create an undue hardship for verified applicants. Eighty-seven percent of recipient 
households subjected to document consistency checks and 80 percent of households 
subjected to follow-up documentation stated they had no difficulty obtaining the 
appropriate document that confirmed their eligibility to receive free or 
reduced-price school meal?. Four percent of the households selected for the 
follow-up documentatioh request reported a problem providipg the document on a 
timely basis. However, informal interviews with determining officials suggest 
strongly that recipients do not comply with the verification requirement until they 
are confronted with the reality that their children's free or reduced-price meals will 



EXHIBIT 7.1: PERCENTAGE OF VERIFIED HOUSEHOLDS REACTING TO VARIOUS VERIFICATION PROCEDURES 
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come to \ah ami. While these figures may somewhat underestimate the difficulty 

recipient households experienced In complying with the verification requirements, 

. j ■ ■ . .■ ' • ■ 

none of the three procedures vyere particularly problematic to applicants. 

7.2 ERROR DETECTION CAPACITY OF QUALITY ASSURANCE PROCEDURES 

Samples for the quality assurance procedures that occur after the meal 
application Is approved can be drawn In many ways.2/ fhls section reports the 
results of the procedures under- two different sample selection methods: random 
selection and selection using the ^error-prone profile (EPP) developed In Phase I of 
IVPP. ' ^ 

The EPP is a method of selecti|^ applications for review with a high probability 
of having significantly mlsreported' Income or family size Information. The EPP 
developed in Phase I of IVPP is very simple and contains only ty/o variables.. Exhibit 
7.2 presents tfije scoring method used and explanation of how the scoring Is used. 
Exhibit 7.3 displays the percentage of sampled applicants' In Phase I of IVPP 
receiving excess benefits by the EPP score as determined analytically^!/ As can. 
'be seen, the higher the score, the greater the likelihood of receiving excess 
benefits. In Phase II, the EPP was tested by comparing the discovered errors rate 
for the various verification procedures both for a random sample and for the subset 
of that sample having EPP scores of 1 or 2. 

/ The verification procedures were implemented In a sample of school districts 
using the IVPP application and school districts ngt using the IVPP application. Type 
of application used was expected to affect significantly the outcome of the 
verification procedures. On the one hand, school districts not using the IVPP 
application had nearly twice the error rate of school districts using the IVPP 
application (See Exhibit 5A). Therefore it is reasonable to expect that verification 
In these school districts would reveal a higher error rate. On the other hand, the* 
IVPP application was specifically designed to support the verification procedures. 
Detailed Income data were collected on the IVPP application that permitted more 
precise verification than was possible In school districts not using the IVPP 
. application. , V 
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EXHIBIT 7.2: APPLICATION-BASED SCORING SYSTEM FOR ERROR-PR'ONE PROFILE 



Step 


....... ^ J. 

« 




A. 


If reported Income 1s wUh1n ^ . 
$120 a month of the free or reduced- 
price eligibility cut-off, wrlte^T on 

line A. Otherwise write ^0.' 

> ■ ■ ■ . ' • 


(A) 


B. 

• ■1 


If reported Income Is wlth^ln $60 , 
a month of the .free or reduced- 
price eligibility cut-off, write on 
Tine B. Otherwise write '0.' 


(B) 


C. 


If the applicant reports receiving 

Food Stamps, write '-1 ' on line C. ^ 

Otherwise write '0. ' 


(C) 


D. 


Sum lines A, 6, and C, and write 
final score on line D. 


(D) 



. Explanation of the EPP Scoring System 

An application Is given one point If It reports an Income w^l thin $120 a 
month of the free or reduced-pMce eligibility cut-off. Another point 1$ 
added If the reported Income Is within $60*of the free or reduced-price 
eligibility cut-off . 'finally, a point Is subtracted If the Applicant 
* reports receiving food stamps. The resulting scale has values ranging 
from -1 ta +2. 

■ •■ ■ '-^ .■: ■ ■ K , 
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EXHIBIT ,7.3: DETECTION RATE OF ERROR-,PRO|JE PROFILE FROM PHASE I OF IVPP 



ErrorrProne . 
Score* 


Percentage Rece1y1hg 
Excess Benefits 




■ 2% 


0 


20% 




40% 


2 


71% 


■ ^ ' ' ■ .'. . 


* 



* Error prone scores were derived statistically by a weighted least 
squares procedure. A linear transformation was performed to preserve * 
the relative weights and produce whole numbers. 



Exhibit 7.4 presents the discovered error rate for the various verification 
procedures tested for sites using the IVPP application. Two cautions are necessary 
before reviewing the exhibit. First, when interpreting results, ,an important 
distinction must be maintained between internal and external e)<periment validity. 
Internal validity addresses the question of whethei)-^the verification procedures 
affected reporting errqr in the specific school districts in which they were tested, " 
External validity addresses the question of. the extent that the Phase II results can 
be generalized to the population. In all pilot sites^ a strict experimental design 
coupled with relatively large sample sizes acts as a strong guarantee of internal 
validity. A nationally representative sample of 24 school districts used the IVPP 
application, therefore results for this group are generalizable,' However, only five 
school districts in Phase II did not use the IVPP application. The resulting sample 
size is toQ small to draw statistically generalizable conclusions. Thus comparison of 
results for non-IVPP application school districts with school districts using the IVPP 
application must be seen as more suggestive than conclusive, ^ 

. Also, the verification procedures are not strictly comparable. The document 
consistency check, by its very nature, could occur only in schools requiring 
documentation with the IVPP application. Therefore this procedure is perhaps 
better understood as an adjunct to the dodumehtation-withrapplication requirement 
rather than a separate procedure. 

The State third-party check of wages (Quality Assurance Procedure VII) 
procedure was conducted for only four school districts, in a single Stated, Moreover, 
this procedure was conducted solely by the evaluations contractor, not the school 
districts, therefore the results for this procedure represent a case study of the 
potential of State third-party check rather than, a nationally represe/itative 
experimental test. 

The first column of figures of Exhibit .7,4 shows that the State thlrd-party 
check had the highest discovered error rate. Although the tape match was 
performed in only four school districts, in all four it outperformed all thp other 
'^procedures. This finding was expected for two reasons,. First, a third-party check 
done by computer precluded the possibility of an applicant withholding or distorting 
information. Second, the State third-party check rairgeted the primary source of 
error: the underreporting of wage income. Phase I of IVPP found that the 
underreporting of wages accounted for 84 percent of excess benefits awarded. This 
fact also helps explain why local welfare third-party checks performed, so poorly. 
Misreporting is concentrated among wage eamer^;^^ 



EXHIBIT 7.4: ERRORS DETECTED BV VERIFICATION PROCEDURES USING RANDQH SELECTION AND SELECTION BY ERROR-PRONE 
PROFILE BY TYPE OF FREE ^ REOUCEO-PRICE SCHOOL HEAL APPLICATION FORM 





. ' DISCOVERED ERROR RATE 


VERIFICATION 
PROCEDURE 


IVPP APPLICATION 1/ 
RANDOM SELECTION EPP SELECTION 


NON IVPP APPLICATION 
■ RANDOM SELECTION EPP SELECTION 


Docunent^Conslstency Check 
(Quality Assurance III) s 


2.Wt 


% 

6.4< 


NA , • 


NA. - 

\ • - ' , ' ■; 


Follow-up Docuinentatlon 
(Quality Assurance IV) 


5.4X 


y 16.41. 


• 

rt 


38.« • 


Parent Telephone Conference 
(Quality Assurance V) 




13.0?; 


• B,9?t 




Local Thlrd-Party Check 2/ 
(Quality Assurance VI) • 


Q.ii 


. 0.3J{ 


' NA 


NA 


■ ■ ". ■' ■' "J A 

btaie Tnird-Party cnecK o/ 
(Quality Assurance VII) 


11.61 . 


'. 35.21,^ 


NA 


NA ' 


Choice of Follow-up Documentation 
or Local Third Party 4/ 


NA 

r 


NA 


11.3X 


34. 8X 

t 



NA » Not Applicable v — » ^ ' 

'V Includes only Incremental effect of /verification procedure, and not tlie effects of the TVPP application 
2/ Third-party checks at the local level were limited verification of welfare and/br food stamp eligibility 



y The" third-party check at the state level repofted here were limited to verification of earned (wage) 
^ income. • . . .. 

if Because the non-IVPP application lacked an adequate release for third-party verification, program 

J permitted to sign an appropriate release or submit 



participants selected ifor verification were 
documentation. 
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In Interpreting the effectiveheas of Procedures III through VII, It is important to 



keep In iTiindthat they were' conducted on a^sample of applicant not all applicants. 
This is in contraai tq'; procedures I (WPP application) and 11 (Docunnehtation with 
Application), which were applied to all apj)licanta. Because procedures III through 
Vn wei» conducted on only 'a 10 percent sample of applicants, their immediate ;' 
" effects were limite^To this sample. As a result, Jhe effebtiJ/6f the procedures .|jh 
oyerall Terror ratest w&^ery small. For example^ using random sample perc?ent 
of the parent telephone conferences resulted in dlsj^yered errors in districts lining 
the jyPP application. The result of these telBpnone conferentfes^^^^^^ to lower the 
oyerall error rate for schools using the procedure. from^ll./ percent to 11.1 percent 
ai.7 X .9 -H (11.7 - 6.1 ) X .1::^ 11.09); ' : V'^-'X ^ - ' 



None of the procedures, when conducted on a random sample of 10 percent, 
resulted in more than 1.2 pSrcent reduction- inthe 11.7 percent'error rate.^ When 
,us^d in corijurrction with e^^ selection, the final error rat^'varied 

'frbma low QfB.Z/percent with StaV^^ of 11.$'' percent 'in thr 

-case of local welfare Smaller samples, such;a,s '^the 3 -percent "sample/ 

recommiSnded by'current ]:egulations,;W inevitably result i'it.sm,alle^ reductio0 ip 
: the final error rates. • • • ' > " i' v V - \ 

Com^iaring the first and second columns .of ^.^^^ the EPP >| 

tmethod of -applicati^ butperfprma randonpt-^selectjonM 

of the fiye j3ropeduj!9S< In . fact, for; - parent, teiephone cdjifepence,--^^ 
documeritatipn, document conaist^ncy^-checki ^and> Stat^.;^t:hira|party check, 'EPPb. 
serection had more th^n twice the .discover.ed' w^^ random seJ^tStioni^ THe^^^^^^ 

exceptidn--l6cal*third-p?rty welfap#Werif|q9tibn--Was expected. The^EPP^ explic|^^ly 
seiecta^nbnwelfare ^ meal recipients for verification. Cohse.quently:jt ifla^ 
♦ incapable of irfentifying welfare recipients with a high probability 'of ^ 



ithe. EI^ performed; equally well in schboli: district?.' nq^^^ 
application. , For all quality assurahce procedures; used irt scfhool dlabricts'vthat didp 
nbt use' the IVPP .application, selection "by EPP was markedly superior to random 
sele^ ^,It is interesting to note that discov^ed error rat;es in the school disltri^ts 
not usiVig IVPP application are-Bignificantly\ in school . di3bicts i|^ the 

ap'p^licajtidn. The* higher discovery rate appears to be^ largely a function' of a| higher 
overall error rate. When the school .district ►Ij^verall' error rate, as discovered by the 
in-home audits, ' .is held statisticaUy coristant,^ no clear pattern exists^ b 
' discovered error '^^te and use of the IVPP application. 




basis-*-that isi cost! 
conducting each ppflj 




..."..-.y.. ■ . ;. ........ ' .... 

ASSURANCE PROCEDURES PERFORMED FOLLOWING 

fAfePROVAp *^ . ' V . * 

X' f ' ' ' -^ " ■ ' ■ ' ■ '' t*^ ' ■ 

Costs assotJiateo With the jprocedures were measured on an average unit cost 

Brhis rneasure excluded all start-up ^nd traihipg costs, 
aU co3t3 to applicanta^^fc^plylng'^Wlth the verification requirements, and ail coats 
to local and state gbvernm^^^ assisting with /third-party checks. These cost 
components -were excluded on the purely pragmatic grounds; that^ reasonable 
Bstimates were i]of available; 'In the Phase H sites, much pf the training and 
start-up costs were absdrbed by the contractor. 'Therefore no reaUv/orld^ estimates 

— : , . : _ ^ ^_ , _ 

are available for, what it would cost a site to implement and monitorMndependently a 
■'■ program of application verification. As a'^ result, the costs' presented are 
optimistically low and - reflect sole'Iy unit costs borne by' sites of conducting each 
.indiifidual yerificatiori.' Appepdix C explains the technical methods ijsed to develop 
the cost estijmates. ; ' t "^ . 



7.3.1' Costs of Quality Assurance Procedures Performed After Applications 
■ Are Approved in School Districts Using, the IVPP Appiication 

Exhibit 7.5 presents the final measures of^erification costs, cost per error, and 

. cost savings ratios of .:each cjf the'-Quallty Assurance Procedures performed" after . 

-IVPP-applications were approved. The foilow-up documentation requirement, tak^s 

an average of A4 minutes to conduct. This requirement needed alrnost twice as ^ 

inuchrsciiool staff time ks any of the other, procedures. Parent telephone^onference 

took '22.7 minutes to corr^plete, and document consistency cheek, and local' welfare 

verificatigri took 17.3 and 10 fninutes^.;^respectively. It is important to keepvHvmind ; 

that the TTieasures'indud&^^nly time required by school district personnel to condtjct ; 

° the verifillions, and not the time spent processing the IVPP application. The^^ time 

devoted by local welfare personnel as well as the time spent by .households arelpot^-s 

, included in these time estimates. . , ' ' 

■■. . ' '\ . '. . • ■ ' ■" V'V ■ ^-^ 

Costs per ver if ibation include time, materials, and indirect costs expended, Jjy 

school districts to perforrri verification, They 'in\:lude only, costs to school dlst^ts 

and do not include costs to applicants or other agertcies involved in verification.^ As 

■ such, the costs to school districts are understated because they do not include cpsts 

• for startru^iand training, which are substantial, nb^ do they include the costs of lost 

opportunities'to school districts; As a partial, adjustment for this.undere?timate, the 

numbers in parentheses in Exhibit 7.5 show costs per verification, including costs of 



V 



'VERIFICATIOH ^ 'il i''yfltt>ER , COST PER ; ; 
PROCEDURE, : ;i|."*':VERIFICATIONl/ ^yEBIFICATlON?/ 



COST PER OISCOVERED ERROt 

mm EPP ; 

SELECTION SELEmON. 



SAVINGS to , J/ 
FEDERAL ■' 
GOVERNICNT 
PER PiSCOVEREO 
ERROR 3/ 



SHVIHGS/CQST RATIO 1/ 



'RANDOH 

SELEniON SELECTION 



ConsUt^cyClieck^^ 17.3 
(Quality Assurance (33.2) , 
Procediirerlll), ' > , 



T 



17.25 



1258.87' 113125 
M99. 



■Oi34 ' • 
(0.18) 



0.78 



it 



Poll«-Up 
Dociiientatloli' ' ^ 

(l)iifllly;fcaii«"~ 
Procedure fV) 



(60.1) 



125.86 . 
($35.24)' . 



1478.99 '$157.71, 
($652.77) ($214.99) w 



$88.31 



0.56 



Parent Conference , Z2./W. - $225.71 $106.26 

(Quality Asainance (38.7) * v($23.54) .' 
Procedure V)' .J,-. ■ v ■ f^'.-.y- 



0.39 



Local Welfare H.,; 
Verification 

(Quallty,Assurance : 
Procedure VI) .■S'lixr: 



"'4 $^«-33: ■ 



$634.29 11480.00 



iao.31 



0.14 



•.06 



^1 Tine in Blnutes for perfirralngiveriflcation 'procedure; excludes time to process IVPP appl Icatlon. Nimbers In parentheses reiiresent time estimates^ 
th»l Include tralnina of school personnel to oerforni verification. These ostlmates are based on school-based verlfltatlons/a 10 percent venincat 



thai Include. training of school personnel to perform verification. These estimates are based on school-based verlfltatlons/a 10 percent venlfication 
SMplei and four hours of training per sdiooI.^'A lower sanpllng rate, sucli as 3 perceoj:. would result In more time (and cost) per verification. 

*2/ ' Costs reflect only Incremental Increases for each procedure ahd exclude costs of processing the IVPP application. Nimbers In parentheses Include 
,:tralnln(j costs, as described Ij the prior note. See Appendix C for a discussion of verlflcitlon costs. 

i-JI *'ihe fedelv saving assunes thall 



I) Jrror Is distributed across the free 4;r|uced-prlcecate^gor1es^;8^^^^ : . 

II) ' 'An average of 18 children per householf partlclpa r I" , 
: |ll||[ Elghty-flve-percent of the ellglbles participate on; any given schoo^^ ' i v Vfi|5r 
: (Iv)^^ Ver If Icajlon Is completed by January 1 of the schobi year, if verification is coiiipleted earllei^l^avlngs'^^ . ' 



il ValueJ iesithan:one Indicate that cpsts of conducting tl« verification exceed savings. Values greater ihan one 'Indicate that savings exifeedcofts. 

. , . 1 ■ " , ■ 'An.: . . . . ■ a'. ' ' . * ' ■ ' ' ^ . ' . 



V--.' - . .............. •,. , .. ...■■-'■vf^f!'»W:;'''' ' . '.-^^■■■'^ \f. ^r- 

i^l^j^l^ personnel. Mpet school dlrtrlcta^that;^'^^ these 

!f Sfe cbatSrVd'HlB , lnd (cabs that ^th^^ Ip tl^!i|^ •resource aflooatlona to 

'.'".•;y '^V;. ■ Ji-jv' -^■v^V''^^-'-^--^"^-/v%»i^V'T>^ ' ■* ■^■'v-.'^; , • ■ . 

'.• V^'/ Therefore it (a apprdprlat^w^^ than absolutely. 

■ A- more detailed dlBCMM^^ 

v'^vTha coat? per VerlflbMlloh;;;^ the 
. time per verlflcatrpri. TheAone rnaj?i<ed that local weffare" verif ipatlons 

; ,>^C08t slightly more to conduct than document' consistency checks , even though they 
. ; take less time to complete. This reveraal is /due toi the fact that ' local third-party . 
^ ■ welfare -checks are commonly ^ cbnductet^ by senior SFA administrative^ ^staff , 
whereas document consistency checks are by. clerical personnel. , 

. , Dividing the cost ' per verlflMtlorv^^^'W presented 
earlier In Exhibit 7.4 results In measures ^ per discovered ^rrqr. Thesie costs 

V are sliown in theVfourth and fifth column^ of Exhibit 7.5. Examination of these costs 
; ' shows 'that, per error discovered, parent telephon^yconfererice is the least expensive 
procedure if random selection Is used. 

.Comparison of the costs per discovered error using error-prone profile selection - 
as opposed to random\selection shows tha^ substah^ can be accrued if the 

erroK^pri^ profile' Is used; the one exception is local welfare verification because ; 
. the error-pr^pe profile expllcit^^ ; 

The fifth column of figures In'^fflm^ 




•.V, «^*«* 



per discovered error, Is presented as 6 benchmark f rdm whlch^xhe^^ 



the procedures, can be evaluated^ . The sitings were noti^ea|^(^(^lirpc^ but 
estimated using several empirically-based assumpticns. First, the pattirns of excess 
.^benefits shown in Exhibit 6.4 reflected the pattern .^t!^ dlscoVerejd err^r, 
\ Approximately three-quarters of all excess benefits went to applicants who received 
free-meal benefits but should have received 'reduced-price benefits^y^ For this: ^oup, 
error disco)/ery savings 'were restricted to the' difference between federal 
. teimbursemfnt^^ for free and reduced-price meals. Next, the savings estimate was 
based ori:5'fijidIngd from both the Phase I arid Phase II surveys: an .average -of 1.8 
. V Students' :waj5 covered by each- jSpplIcation^ ^and studehts' on the average ate free or 
reduced-price meals 85 percent'pf the time they are offered. Finally, the.f|fectlve' 
^ date of eligibility changes bas^d or^ verification* procedures was as3urri^|to be 
■ 5 January 1, or about four* months after the start, of the school year .if ^Eariier 



ERIC"-/ , -.^Z;/; .; ^-... ./-v.^' 



verification would ppodii:8 ^ro^ leBoej)/ wVh^^ ^ 

CoUectlvelyr these aqoumptlons prodUci|^ an satln^fcted aavlnga per discovered error ^ - 

$8BJ3.5/:;--;';::\;^ . v-,||y:'''^ ■■ ;■. V;;- ^ : ■ " ^ -/^ ^- ■ - .-■ ■' . 

The last two, columns of . Exfilblt 7.5 are simply the ratio of savings per , 
discovered error to the ^^^^^^^ to cdsts borne by sohooi dlatriqts. A 

'ratio greater th^rV one Indicates aiavi^^ of less than one rneana : 

that the cost Qf^^^^^^^ the .federal government. For 

example, the uppiar left-hand figure of .34 me^ns that for every dollar spent By local .^. ^ 
school districts on document consistency check verificationsi $.34 was saved by th^ 
federal government and none by school districts. * 

Note that aeverpi factors complicate the. savings dost analysi 
sayings occur to different governmental units. All sayings accrue to the > 

federal government. Moreover, cost savings for the federal government represent 
revenue loss for/local school districts. As si iresuit, when judged strjictly from the 

^fqder^i perspectivei an;^ quality assurance procedure is likely to be oost-effectiye 
because requiring the procedure wqui^J Involve no ^(^ to the federal 

government and iia likely to result in at least nominal cost savings. Corvyersely, from 
the school district?^ perspective, no procedure; could theoretically be cost-effective 

' since- any quality assurance efforts are likely to involve additional school district 
coats. Jf the procedure discovers or deteijs misreportin^, the monetary e/fect on. 
ideal school districts wou^ ' - ^ 

' therefore, to , rely solely on savings-cost ratios to evaluate tl^fe^ 
would be misie^ipg. However, careful analysis of; the costs and sa^gi 
empirical Bxpiic^t'ldn of these! issues^ and could form the foundationfqfi 
syatemA^hat would, make qual^^ botj 
govefhmeht and ioqal school distyB^.tsy Altfernat sayings-cost; ratioa can be 

3eensfr^\J^^^ it makes iitye difference whether the 

i^l^^ costs accrue on the feder^^^^^ ' . ^^ 

All the savings-cost Tf^^ verification procedures in scihooi districts udna^^^J^^ 
••tlp\IVPP application ^^^^ Therefore 4t costs more to conduct the^ ;^^ 

Verifications than is^saypd ;tKrough error disco've%^. ,If costs to school districts of 
tralP^gjJ start-up, mp^ costs to applicants and state art^local v 

governments are inclucled 4n the figures, the savings-cost ratios ^.wouid be much 
lower.-- ■■ 'rf.' ' X ■ / . 




savi^ 
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: the aemple alze of achool dlabrlats not .gaing the IVRP qp^llc^tlon vyaa too small • 
and the acroBa-fCilte variance jh cpsta too large to stabilize atab^tlcally adequate 
eatlmetea of verification coata, The ; great aoroaa-alte variation ^ ^^^1^^^^^^^^ 

^ verification for school dlatricta nat,M8^^^ IVPP application was primarily a reB\^ 
of abme altegriri thla group conducting yerlfloationa centran^ at the achool dla^lcfi . 

, level ^nd othora; conducting the verlflcatlona at the Individual school.^ 
verification procedurea were much more coat-effective when condi!(cted centfally at 

r ' the dlatrlct level. The following section dlacusaea this Important distinction. 

^- ■ ' " ■ ■ \ . , ■ . . , ■ ..' ■ ' . - • , ■ ' ' 

7.3.2 The Costa of Centralized and Decentralized Veiriflcatlon , • 

» With the e>tception ofone siteV local third.-^^^ ghecks were conducted 

• by SFA personnel, not by, personnel In;^^ the other ^ 
> procedures, the. pattern was reversed. With few exceptions, parent conferences, 

* document consistency checlis, and follow-up'\ documentation requests were all 

. ' ■ ■ , ■ • . ■ ■ . . ' ' ■ , . ■ • , • .■ . 

conducted by Individual .school personnel and, with little Involvement by central 

district personnel. - ^ ' ' ; v ^ 

This finding re|l0feted the :genera^a|ructure of ^program application ^^rocessing 
in school dil^^taif^it^^ majdirlty^ ci^^^^^ 

district, meai|6eneVk^^ dlstril^(j1:e^ processed : in^-^ 

individual schooC There f|c^;^i):^^^ the majority of sites In '^^^^^ 

' Phase il 3amj^l^.ch the procedures at the school^eyel; nrwst.s^^ 

: !^ distr^^ resources^ to undertake^^ 

^ Local third-party W^^^ were 'an exception only because of the nec of ^? 

*:V^ centralized coordlHatlBn with local ' Welfare , agencies. ■ For this ; procedure, the; 
evaluation c6c|tr actor provided support services by creating computerized lists^^j . 
aoDlicant households to be verified, ^he evaluation contractor also worked dlredjiiiy^^^v^ 
with many local agencies to explain the verlficati6n'''needi3' of the experimentar 
.sites. It is doubtful that many of the ichool districts could have completed this 

*/-N4)rocedure on a timely bOTis without such support. . , * .^ ^^ 

^ The 'decentralization a/ assurance ' implementation, which appears 

liferent 4n the cun-ent strqcture of the public education syatem, resulted- in 
8ul?3tantiaf diseconomies of ad^le-that is, the relatively low 'numbeig|||e^ 
per school generated high .draining and start-up costs relative 'to th^ costs of ^ 
performing the actual | verifications. Vfiph the Phase II 10 -gisrc^t 
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^/wlflcatlan aamplo^ pn: average of tewBr than 15 .appUoatlona ware varitled. per 

flbhooU Trailing of school ; perapnner^ conduct the prooeclupap pequlred , 

apprd)<lmately :A/ hours eaoli^ 

provided, The rewU //ae that more time waa apent In training arid assistance than In 
actually conducting the verifications. ' ' 

' A second . Important diseconomy of scale Involved iabor costs. When conducted 
'at the sohobl level, 56 percent 0^ principals and A4 percent 

all other labor categories— primarily secretarial and clerical. The relatively high 

salaries of prlhcipala resulted In high labor dosts'for verlfloatlorit ; ; 

.. ' V ';. -^i'-.y/ " ' - , ' ■■■ 

. The > third dlaecqnomy time required , to learn, b 

procedure. Generally It takes an Individual longer to perform a taak the first time 

than for subsequent performances. The small number of Verifications per school 

prevented such routlrilflitijpr^ tirruS;:^ School 

level was uhnecessarlli^igh. - ^^^^^ :i^rr:-.yr / 

j^l^^ incomeoVerlflcatron can be 

sre^ of three '^^perlmental sites located In the samp 

State. AH the^ sites were in largp; cities with similar defpographic a 

V(^donomic characterise ugJeS' n^lly Identical app^^ Howevepf 

these were riot IVPR ""ajSl^c^ they more approximated the USDA-recommehded 
application for the 1982-83 schodl Year.,^^e three schodr districtia 'had nearly : 
ident ical ' 'program terror tatie^f and all ||free implemente^^^the same set of quality 
. 'assurance. prdceduref3.jS/ ■ f^oWevel^r la one!naf the procedurea were 

^.^onducted at the 3bhoQ|^|.yel. ^^^^ tjiie remf|ining school districts, yerlflcations^were > 
l^conduc ted centrally by si^A'cierical personnel were 

• dramatically, different;^-^^^^^ r 'M^ 'Z 

.^v= Exhibit 7.6 64?mpare8- thq^,t^^lffle^^ verification for 

llij^esej t^ dis±rictS5. In the ^Ibit,: schddf d 27 conducted 

:;Ai^tralitz^ and school district 26 wasCj^ ^ 

^|^;,|d^^ figu)p^ represent^j^y unit measures >qf c^^ theV actual 

i^!|^V^rificat|c^ times and costs hawj^ been excluded. ; Because of 

, an-expohehttjll^ jndlyidijiala^cpnduct^^^^ in. schqol ^ 

* disti^t 2<S/ the-^^ are much lar^^ those 
Shown In the.^xhifcitv1^^ were measured' 
and Apperidix for; more det^ thesb districts.) The 
enormous din^npes In 4^ are indicative - of the inherent 
inefficiegcea bl^oc^dubtin at^'the^ school level. ^ ' 



kkmi 7,6;* COMPARISON OF THE PER VERlFlCATiON TIME AND COST'! QF COMPLETIN^^^^^ 
.^'^ ' ' VERIFICATION IN 3 MATCHED SFAS THAT DID NOT USE THE IVPP APP1^0^ 
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Parent Telephone 
Conference 



Follow-up Documentation FolloW'-up Documantatlon 
Request Reqi^est OR Third-Party 

'4^' Verification 




Parent Telephone 
Conference 



Follow-up Documentation Follow-up Oocunierttatton 
Request Request OR^Ird-Party 

Verification- 



A-r Verification Performed Central ly, by the SFA \ 
.B-»- Verification' Performed Centrally by the SFA . 



District 
District 

District C— "Verification Performed by Offltlal In Each School Building 
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BEST CO?Y AVAWCIE 



7.ft : EFFEdtS or QUALITY ASSURANCE PROCEDURES ON PROQRAiftf ; 

PARTICIPATION ' ^ - ■ - ' 

/ The moat obvloua effoot the prooedurea had on ppagram parUdpatlcin wap thut 

, ^^ppUoant8' found to be Ineligible were removed from the program. For Qeverefof 

the prooeduroai thei effecta on participation were limited to the removal of 

ineligible peraonei 'Looal thlr^^party welfare cheok} State third-party wage oheoka,, 

and document conalatency check prooedurea were geiierally conducted without 

direct applicant knowledge or cooperatlonf Thoaa procedureo.'affected participation 

only if errora were dlacovered and beneflta adjuated, » 

The three' other verification procedures 'teat^d— parent telephone conference, 
foUowrup documentBtlon reciulrement, and choice of third-party verification or 
follow-up docuiTt^erttatj^lop-'all required appjiicant cooperat^onf Failure to cooperate 
resulted ln_|he denial of meal benefits. In the case of parent telephone conferences, 
nearly i^^pe^^ achieved, ,Pply 1 perd^ent of applicants 

contactedpfoc a parent telephone conferenq^ refg^.d to. c 
-frpm the program. The num.bsr of noncoopMSflng applicaht^v.in the 'Safrjplev^^ 
small to develop.a meanirigfUl eatimate of what proportion was VUgible for b^nefi^.j^-^ 

Compared with parent telephone conference, a much higher percentage of 
.. applicants did not^bmply follow-up dopuirfentation request. In sites using 

the IVPP application, the nbintirdop^etf^^ was 8.3 percent- and the bodperatfpn 

rate was 91.7 oercent. . When given the option of providing a document or signing a 
release^allowi^^'hird-party income verification, 14 percent failed to comply and 86 
percent cooperated. (Note that this?' procedure was only used in ^iX^ij^not e 
the- lYPP applications.) 

Althou^' most applicants did cooperate, Jt- was important t^^^ 
applicants who failed to bdoperate. If the \p,i(dcedure was effectr\i^et¥n^^^^^ 
ineligibles from applying, then most of thes^eK^fppirQ^^^ Based' 
. : V upon the in-home , audits; most (7.1% out ofA^i3%^^ 

^ / benefits. If barrier is defined as the^pdrti^nt d/nonc6bperating applicants who are ■ 
.,truely eligible, the.h this procedure has a stTong barrier effecL Five out of six of 
the noncooperatiog/japplicants wer^^ truely eligible to; receive benefits* If^barrier?i3 
defined as the percent of all applicants subjected to the procedure wh^'^^fail xt^^^^^ 
compl>(^wand are^lruely eU then this procedure has a; mcdqst barr^^^ 
Overall, seven out of every 100 applicants who were ^asked-to submit docum^nti? 
were eligible, but were barred -from the program because' they Tail^ virith 
j^|;^fei%|»que3t. •. - " ^ ; • /.V.-;'-;.,/ ••• *' \ 



END NOTES 



1/ Questions ^bout thlrd-party quality aeBuranoe prooisdurflB wars not asUecl 
bQoauae ppiaaram partlolpanta waro opnteioted bnly If ' the reeulfco of the 
third-party mWloatlon raveBlad mlw 

eligibility otatuo. Only a vdry smaU propartlon of tha verified houaaholda were, 
found to have mlarepqrted due to the faot that food atamp fi(llglbUlty and AFpC 
beneflta were verified rather than wage Incorhe. . 

2/ The quality aaauranoe procedurea dlflcOoaed In thia Qhaptar are dealgned to 
" detect error among approved applloanta. Quality aqaurance' procedurea 
dlaouflfled In Chaptera ? and 6 were dealgned to deter mlaf^eprtlrig. v,;.$,;„^ 

3/ The Error-Prone Profile wea only uaed analytlcaUy, It was hot uaed. by'^ the 
experimental school dlatrlcta to aelect rBclplanta for .verification 'beo|i^ 



would have biased the reaulta of the study. ^ 



4/ A fout-month period was used because .It cloaely approxlnria period of 

• time under interlrn verification r&gulatlons for completing Verification. 

5/ It should be ramarked how relatively low this figure Is* One telling contraat is 
the saving? per discovered errpr In Social Security Dlaablllty Allowances^ For a 
recently completed Brror-detection system that waa dealgned by the evaluatioh 

■ contractor for the Social Security Adminlatratlon's Office ^qf and 
Apjieais, average expected aavjnga per discovered erf^^ were, $30,000, or 340 
' 'times as much as the savings per discovered error In the school lunch program. 

6/ These .procedures were very 8^^ those tested In sites using the IVPP 

^ applibatlonv|^ All , proceodurea^ . cotjjdQcted after ^ttie appl^lpn:: tied been, 
ap'prtved. 1^^^ folfow-up docjurnentatlon "requ^^^^ 

telephone conffience, ^and an- .applicant-exercised option' of follow-up 
documentation or third-party checkV • / ' - ; ; 



' SUMMARY^ND CONCLUSIONS 
•. ■. ' ■. . . . • ■ .. ■ '•• ./ . : ■ . ; : .. ■ . . . 

^ ^ chapter aummerlssMthe f^^ qonoluqlona of Phase 11 of the Inoome 

VerlflcfliMon Pilot ProjeotvlSe^ erpor-roduclng and error-preventing quality 
asaurahQe' prooedureo for theTfree and reduoed-prlce school meal programa were 
tested In a national sample of school districts. The summary provides) 

- A review of the problertn of error In the free and reduoed-prlce;5r$chool 

y meal programs and factors that will shape the quality assur^ce 
procedures Intended to remedy the problem , 

^^ if ; --'M A description of the seven quality assurance procedures that were 
implemented on a pilot basis and methods used to evaluate their 
feasibility, costs, and effects 

• A iBurnmary of findings on the. feasibility, costs, and effects of each 
procedure, and ^ 

• A discu3sldn^o?^l:^e conciu^ton3 reached about the results and (mplications 
^ " ofPhase^Il^fiyPP. / '^ 

^ 8J QUALITY ASSURANcie IN THEFREE AND REDUCED^^^^ ' 

' ■ -PROGRAMS^ ^■:m'^-'r: ; ■ ■ • > ■ ■ ■ . : > :. ^ ^ . ■ ■, . - ^C-' :, . 

The problem of mlsreporting income and household - sizp to obtain free'' or 
reduced-price schiDol meals has beerh;documenfied' in a number of 
ySDA^ Correcting the . problem through :^;t assurance^/ 
' procedures was authorized by the Congress in with the passage of Public 
,97--35. As this repdH demonstrates, successful quaUty asaura 

school districts that participate in the free and reduced-price school meal programs^ ' 
must, recognize and accommodate^ factors that characterize the pvo^v^j^^^^g^j^ 
factors are discussed briefly. ^ ! 

^ Within *the public education system, there is a lacki;^^|^^perieni||^^ 
^uaUty assurance. for the s^^^^ meal programs as applied- to the vMM|s|iJ^»i^ 
'/.applicant household income and size. Other programs, such as food stamps, haye a 
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V relatively lc)ng§r: hlftt9Py :of using quality iBB^pihQS proeeflupes fcO Mduoe f^jt^^^j^l 
mm, Repeated itudlsa and veplflQatlon demQnstratlQm have been aonduQ ted' to 
reduoe tHf mlareportlng of ■ eirgllJlllty-pelatBd Infopmatlbn. Unlike .'faod 8tam|38, 
there le no aclmlnlatpatlve atpuotupe Pf e^ In publlQvCand' pplvate) pplmary. and 
aeeohdepy eduoetlon ayatemfl thB^^^^^^ to Implement AqyilUy aBBUPan^^^ 

pppcl'edurQB, 3ohool d latplots must ,otapt wlthlut benefit of the experlentae <and 
training- that fop'd atampa and other programa have developed, In addition, there la 

' no apQQlflo aouroB of local sol]qol dlatrlot funding dealgnated for quality aaauranoe 
(although program regulatlohB allow atatea to ueo admlniatratN - funda for 

, ;verlflQatlon If done at the atate level). At the SFA level, regular program meal 
relmburaementa may be applied to ^erlfloatlon, although no fun'da are apeolfloaUy , 

. earmarked for verltloatlont Jf quality aoouranoo prooeduroa are to /bo Implomantecl 
diligent lyV funding from or otata aouroea or dhangea In the uae of 0Kl8t|ng 

funds may be neceaaaryt * 

^^V . The potential for pavings resulting from quality aaaurance In the f^ree and 
' reduced-price achool meal programa la Inherently amalU The average annual aavlnga 
irt federal funds for ea^^ determined to have mlareported la about $88, 

This amount la a fractl^*df the aavlngJper beneficiary achieved through quality 
aaaurance In other federally aupported aaalatance* programa. Moreover, achool 
dlstrlctB do not directly benefit from the saVlngS because , it Is likely that they would 
be responsible for-the cost of the quality assurance procedure but would" not share Iri 
the savings./ Unless costs were recovered from the Ineligible mlsreportlng 
households, 'School distrlota would lose the feder|pi^]rTpi^rsements that would have 
been used to offset the fixed coats of operating thefriegilpfogram.i/ 



8.2 QUALITY ASSURANCE METHODS IMPLEMENTED IN PHASE II AND THE 

*»/lETHGDS USED'TG TEST THEM . ^ |f t * : ^ 

The seven quality assur^Oe procedyjjes ]iil|leme^^^ pilot^tudy ,}<»/e^ - ' 

I, A Reviaed Apalicatlon Form dealgned to deter miareporting and to provide 
* data needed to aupport the legal and informational requirements of 

aubaequent verification procedurea, (The application ia referred to aa the . ^ 
IVPP appUcation.) ^ ' / / K . 
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TSpertsf on . the • ippllQaU It Bhould tse not#r)hat thia ppooedurs 
v' r- - Involved only submitting thi doBum^^ tham-wlth fcl^e 

: appllqat orif and as fl.Moh vyas Intanded to detar mlspaportlng, 

• ' Both theoa prQQadMpas, wsDS implem^^ the start of the* sbhool yeap whan 
#ii;;9pplloatlano are riubmlttddVa^^ allglblllty. ^hua. thay -apa 

pplmapily datapponoa pp6aadupB8. in mqit sbhQo) ;dlftp(i5^^^^^^ two ppoaaduraa 
were taated by^'aohOQl t^^^ (a.gi, pplnolpali? an4 9oh9Ql oeoreiapleii) and 

lnvalVocl,^U*tuidentW' -^^^^^^^^ ' : ^■ 

a?;^nv|||fidltidnai ^^uaUty aa^ Implemented lalsep In the . ,^ 

I appHig|tl:pnf had, been iapppoved* fhua . thay. were prlrnapUy 

H^iW pi^afidurea, The pro . v 




DoQim^ ConoiatenQV Gheok following apppovai fdr benoflta : to' 
^^'^¥^^Sk.^i<''9'''niino If . ■ the ■■ .dociument Initially aubmltted with the •applioatlort ' 
' " corroborated Information on 




>P^^'*^ IV, DoGumentatlon After Applfeatldn with Conslotency Chedk to obtalri 
^ ; documentary proof pf Income *ahd other benefit^ proaraKO eligibility 

..^ I reported on the application; .(This ^procedure la aloo ^referred to aa 
follow-up docurnontatlon requok.) ^ 



V. 



V# Parent Telephone Conferenoe following approval of benefits to obtain all 

\ application Items In o]p4er to verify IncomeVw^^ family size Information 
t originally reported 5n trie'appllcatlon. ' ^, ' , 

VI. Local Third-F^arty Checks (e.q.» welfare office data exchange) following ^ 
i- approval of benefits to verify 'earned Income or Income maintenance . ^ 

progreyn eilgiblUty information. ; ^ .' ^ ' ' 

■ . ....■^^>\-'. ■ ■■■ : ■■■ ; ■ 

VII. state : Third-Par^ Checks (Lom computer wage, tape match) following 
• approval .ot"b6nefitar to vqfify earned income : or income maintenance 
" prqgram eUgibility infprnriation, 1 



o 



: Document consisteocy check, documentation after application with consistency 
?-::bh'eck, (or follow-up documentation), and parent telephqne conference were primarily 

cGtnducted by school building official^. Local third-party checks were conducted by 

• ■ C ■ . ■ ■ ■ .■■ ■ ■ ^ ■. ■ 'w ■ ■ 

•Kilstrict personnel, and state third-party chetJks were conducted by , the evaluation 

contractor. All ifive5i>1|^^^ in contrast to" the first two procedures^ 

were 'tested on a sample of students in- each^school. ' *• 

..... • V- ■ V ^ T\ ■ , « - . ■ • 
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There was, In. addition^^^n -Importjarib variation** In /he Implementation ofj- \ 
procedures. In a select number of large achool.dlstrlcts, the IVPP application Was 
not used. Rather the applications developed by the state or school districts 
themselves were used. In these districts the verification proced)Jrea. included 
foUow'-up documentation request, parent telephone conference, and applicant option 
of follow-up documentation or local third-party contact. > 

The implementation and evaluation of the quality assurance procedures were 
accompUshed using an experirfientaLdesign. This design featured: ' • . ; . 

• Matching experimental and^control states; \ ; j - ^ 

• , Sampling experinrontal school-districts (where' the verification prcfc'edures / 

were implemented) in the experimental states and matched control school / 
. districts in both the experimental and control states; and • * v ./ 

• Statistical matching and random assignment of "all schools in th^ 
experimental, school districts into experimental groups. / 

matching variables included poverty level, locatiof^ and school enrollment. A 
tai of 16 staters— 8 experimental and 8 control— and a nationallyrjapresen^tative' ^ 
pie of 120 schodl districts were selected tcLparticipate in^the pilot project:: 30 




[experimental sites and 90 control sites. Through attrition during the course of the 

t, the^inal study sample included 29 piloLsites and 85 control sites. ^ 

: - ^ ■ ■ . ■ ■ ■ ' ' . 7 ■ 

^rall, a total of 755 school buildings with a total enrollment of over 379,400 

students partigipated in the experimental school districts. Over /78,600 IVPP 

applications wfere processed, of which 33 percent included a document:. Ten percent' 

of the applicants in the experimental sites were subjected to one form of 

verification performed at the schodl level— parent tetephone conference, follow-up 

documentation request and consistency check, dr document consistency . check. A 

second 5 percent samplef^ from the first sample and ajiqther 5 percent sample not in 

the original sample were subjected to either a local or state third-party check.2/ 

■ * ■ . ■ '■•/ ■ ; ■ 7 - ' ■ ■ 

The procedures were evaluated on the basFs of data from IVPP applications and 

non-IVPP applications, detailed outcomie reports completed by the School officials 

who actually conducted the procedures, questionnaires complete^ by school and SFA 

personnel, and in-home income audits, with a random sample of 1,810 applicants in 

IS^pildt sites. Also, questionnaires we're completed by each of the matched control 

school districts. The resulting data were analyzed, and the results are summarized 

here. / / 
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Ogtcorries - of the Income verification dejaend, in / large paVb, ,on 'which 
' application^ are selected fqr verlffeation. The quality .aaaurance procedures that 
opcurred after applicatiqns were prficesse'd> and approved were analyzed uslgg tv^o 
different sielection methods: , random selpction and selection using an error-prone 
profile (EPP). The EPP Is a method of selecting applications that have -li high 
probability of ^nificant mlsreporting errors— errors that result in the Receipt of 
meaibeneffts to which the applicant is not antitledr Developed as parjij-of Phase l of 
"IVPP, the '^P/itargets applications that ^have reported income near tee eligibility 
cut-dff point arid that'report the household does.not receive food stamiTjo^nefits. j- 

8,3 PRINCIPAL QU^TY ASSURANCE PROCED^ 

♦ .■ ' ' *' • * ■ ^ ' ■ ■ 

The principal finings of. the test af each- of ^the seven quality assuranpe 

procedures ^e provided. Exhibit 8.1 summarizes many of the findings described in 

previous chapters, . - * . ' * { 

Quality Assurance Procedure I; IVPP' Application / ^ . ' ^ ' 

Among schools that used the IVPP application, an average of 150 apijlications 
were received, of which 39, or 26 perceht, were returned Helcause.they lac lied the 
Spcial Security number^ of all adults, household income, or appropriate signatures. 
However, an^verage of less than one apphcation pgr school was denied becauJe-the* 
application wasi/otherwise incomplete or not returned within a reasbrtsble time. 

School ' ^ersolTner spent an average of 16 minutes processing each IVPP 
application, or 42.5 hpurs fpr all applications received>by the school. \JFhe averagig'* 
time required to process a single application varied from 6 minutes to<-22 minutes. 
The average estimated cost to school districts of processing ^n IVPP application was 
$4.53, while the 'estimated cost of processing a non-IVPP application wa^$3.05. The 
difference fn co>t is attributable to increased time ra^r 'than different Jiypes of- 
schdol staff processing applications. These co^ts do not indlude costs borne by the ^ 
applibant, nor do they include training costs. ' * ^ 

School perspnnel^in sites uMng the IVPP application: reported a.high degree /of 

contact with applicant households. Sixty percent of these officials repc^rted at le^st 

one complaint, about the IVPP application. Forty-four percent of the determining 

officials reported that at least one parent did not fully understand the application, 

\^ . . • ' - . '• ■ ^ 

and 16 percent reported that at least ' one parent did not want to complete the 

application^ In an independent survey of applicants, few households reported any 

tangible difficulties in^completing the IVPP applicsftion. 
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IVPP |tipl1ciitlon 



'H Oocumentallon with ' 

nr. -;Oocyiiiej)t Consistency" ■ 
1 Jliecit' ■ 

IV' ■ • Docuptjtlon ftfter 
. ilppllcjtlon with 
, 'Consistency Check 

° i<|l • Parent letephone 
Conference 



.; Party Contact 



state T|)1ril- 
Party Contact 
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lift PER : COST PER ERROR 
VERIFIC/ITION ' VERIFICATIfl|l^3/fl/. RATES/ 



Deterrence | Ifiinln. • ^J4.53 
Peterrence. : 22ip1ii.l/. JUO 



HA 



'.Oftectlon 
Detection 

Detection 
Detection 



«.iiiiln# • J25.86' 

i. ■ . 

in Ml m\ 

lO.Omln.i/ .m 

..^...L-ji^.^ 



Detection 




JAVIHGS/COST RIITIO 
WNDllHSElECTloi EPP SELECTION 



. 0.56 



0.39 



0.14 



y Includes time ';d process the application) as weiras' obtain documentation. 
i' Excludes time to process the application, only includes time to perform each detection procedure. 



'1/ Deterrence procedures are applied to aTl applicants, therefore total costs to school districts are greater than, total costs for detection procedures - : 
' . . ; , applied' to a sample of applicartts, 1 ' ■ 
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applied' to a sample of applicants 
il , Costs do not Include costs for set-up and training, which may be subs antlal (see Tahle 7,5), ; ' v 

11 . For procedures 11! -'vn the figures' sliown are theittiii discovered error rate. The discovered error rate that can be obtained with' tlie error-prone 
;■ profile Is shoiln In parentheses. 



ERIC 



loo- 



; In sdhooi dl^trlcta using the IVPP;fpp.|iqatla^^^^^^^^ program 
,partlQlparita received 'beneflta in e><qefla"/f /th68|| to wh thay were legally 
ehtltted. "of theae partlclpqnta, 8,7 pefeent,recelvi?:d fpe^-^^ 
entitled to reduced-price beneflta. Another three .percent. of ^ the appiicanta- were 
Ineligible^ to receive aiiy beneflta. Among this group,' one-half., of ope percent 
'received free ;meal, beneflta but vyere\not entitled :td \qn^ 2,5 percent 

^ecejved reduced-price benefits but were n^t entitled to any bepefifed. In contrast, 
23.7 percent of the applicants in Phage I 
received excess benefits.' Phase I of IVPP, 
;IVPP applicatlcy, found an excess bepefitlrate pf 17,4 percent In 1981-82; the 
Nationa.r Evaluation of School Nutrition Programs found a\21,8 perbent error -jrate in 
1981-82," and the USDA Office of the Inspector General found a 28,8 percent rate In 
1980*81, Thus four samples not using, the IVPP application had veri|ied erron rates 
of 50 to 250 percent nigher than IVPP application sample prror r^ite, MordfZwer, the 
error rate among ineligibles using the IVPP apRlication was between one-third and 
one-fifth of the. error rate among ineliglbles^in the four samples not. using the IVPP 
application. : \ ^ 



■ 



SFAs;' not using the IVPP ap)Dlicat;loh 
which used .a rudimentary form' of the 



These comparisons constitute stjlxjjl^ evidence that the I^P application 
successfully reduced misreporting, especially among^iheligibles. Because the Phase 
.II design did not include a statisticajlyj matched control group with a known error 
rate, no definitive estimate of the magnitude of error p^uction attributed to the 
IVPP application was possib 1^.2/ However,^ it is evident that the IVPP application 
does reduce error substantially at the start of the school year. ' 

Given the application's effectiveness in reducing the number of the ineligibles 
receiving benefits, a decrease in program jDarticipation- is to be- expected. School 
districts using the I VRP had a 5;8^percent lower rate. of program participatibni than 
matched contrdl sites. This estimate j& subject to significant sampling variance, and 
it IS not k^oVi/n how 'much of the reduced participation is attributable to d^errence 
arid how nrVuch, if any, is due to the application qreating a barrier to households who 
. are eligible for program participation, ^ 



Quality Aaeuranoe RrooeduVe II Inoome Dooum IVPP , 

Applloatlon *^ y v '\ 

v^; . When rev law Ing.and ojpprgvlrjg the oppllcatlon wlth'dQoumentatIqn> an averaae^- 
of (65 applloatlona reoolved (46 paroentV ware re^^^^ each qchobl beoauQa th\ " ; ■ 
'tecompanylng doQumentatlon, \^^efl9;lnQoprect,'.or mlaBlngi ' In eplte of thio apparently' 
hl^h number, ' an averag^ of only two applloatlona per aohool (1,4 percept) were 
d^lad bjeneflts bec^uae the appircflnt reluaBci to provide the required ddcuiDint, An / ' 
average of leaa than one applloatlon per aohool was denied bacauae thq applicant waa/ 
unable to provide theTBqulred document. / 

Adding the document to the IVPP application aa a condition for applying for . 
meal beneflta Increased the tlm^ required to review and approve an IVPP application Y . 
by an ayerage^of 6/ minutes., Thia repreadnts a 37 percent Increase In time* Thla 
increase variety froml.4 minutes to 21 minutes. Thus In a typlpal school where this 
.application requirement was In effect, I ah average of 61 hours was required td 
review and approve an average of 140 school fneal applications. The^ average . . 
estimated' cost to school districts of processing a document and ttie IVPP application 
was n$67l07^dr 35~^b rcehTli ighFr"~^.Ra^^ 

nearly twice the cost (97 percent). of processing non-^PP "applications. ^ ; 

. , \. , i ^ . . . - • , . • 

Seventy-eigh^' percent of all determining officials who processed the'^ • 
documentation with IVPP application reported that they received at legst one 
complaint from applicants about the documentation requirement. However; in-hoi^ . 
audits conducted. With' households subjected ' to t^^^ with IVPP ^ 

application reqafremant revealed that 8^^ could not recall any 

specific difficulty in /obtaining the required documentation. Six ancj one-half . . 
percent of ithese houMho Ids reported that they had to cojitact a thJrd-parfy agency 
(e.g., welfare office,v^mployer) to^ copy of the document. Eighty^three 

percent of these' applicants reported that although they recalled the requiremefrrtL^^ 
did not raise any significant concern or problem for them. / ^^^^5^ 

The documentation with IVPP application requirement appears to have had a 
relatively srtli^ll effect on misreporting. Applicant/ required ^ to *^ submit 
documentation with the IVPP 'application had a slightly lower excess benefit rate 
than applicants not subject to the requirement (9.4 percent cornpared. to 11.7 
percent). < * / ^ k ■ ' ' 
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In all 3Qhoola-'reqij|lrlng dqoumentetldni wltb ^tha lypP. appUoatlon, aohoql lunoh 
prqgrom^^ pEirtlQlpatlQn w8fl low^r Ir^^ ''dopumanta^lpn" ,' 'sohpolft /1.tiaa ' ■ in- 
'•nondoQurnar)iatloti'* achobla. iPn tha avewgej aohoolij . requiring rjoyumantot^ with 
'the IVPP appHpatlon ,Bxpariano9cl> on 9,7, perQent idwer participation rate than 
achoola not roqulrjng it. The- overall ef^ot of thBldpoumehtatlon raqulfamant on . 
" participation ranged from 2.2 perc^nt at^thi^ M:anlnMl7,0,percent at the high end. * 

Deterrence—redugtlon of ml9P'eportrng"l3-eatlrn^ted to ..•account for - only 1.4 
, peroent of, the i 8.7 percent reductlop In |partlqipatlqn aaaoclated With the r 
doc|imentatlon requlrernqj^l, the remaining 7i3 jper^^ la , due to barrier effects. ^ 
th||mean8 that for, every. Inillglbie h.ouVelj'old deterred frorin m|l9reportlhg because. . 
oifWpfj!clocu with IVPP appUdatlon; requirement more than five eligible " 

hoU3ehqld%;i^,d not participate In the jarograrb^ ; ^ \ ; " * 

Quality Aasurance Procedure 11?;,,. bocument/|cbn3i3tehcy^Check : 

to schools requlhng^ lncdm? d th^ 

incoi^p0<^document was compared with Information on the application for a samplezof 

•^-^ . ' ■ / ,: . ... ■ ■ y • . ' ■ ,^ ;• • • . ;;] 

applicij&i^ts, Thi's procedure tqok place after approval of.^the. application. The intent - 
of the' procedure wa^ detect errors ^on the^^ypP^^pplication• The document 
consistency ch^clc^pcedure detected a ZiB^ei^nt i9xces3 b^ni^flt i;ate when ^ , j 
applications were«>stfl|cted^^ random and 6,4'^rcent when selected, Using th^ EPP» • 

I The average/ ^fci^m'erit cdri^^^ checlc took 11 minutes and cost $4,76* 

However, ^^^^.t^le 'ti^^ cost of obtaining the. document is added, these numbers 
rose to l^i^^M^^and $7.2^ per^ consistency check'. The resulting, totais .indicate 
that for ^el^y 3^0 liar spent b>^. SFAs; on •obtaining docUrri^ents and conducting 
C0nsi%tency\^checks, the federal :goveri^ sav^ $0.34 if random selection was 
used and $0.78 if EPP selection^was used. , . . 

Quality ' Assiffahce- Procedure IV: y Documentatibn Following Application with 
' Cjjjrisistefj^Cheek Z ' ' , ^ ■* • ■ • ■/ , ^ ' ^ 

V, In pr^entT^^he ' results of this follqw-up docuitieiitation procedure, school 
districts that used- the IVPP application' are contrasted with school districts that did 
nqt^use the IVPP '^plication. . / ^ ^ 

Desj^te extk^ive training, tSehnical assistance, and m^onitoring efforts,. 31 . 
percent ofJ^hCf^^enfs for thiS'procedure reported never having' ratjeived ^ , 

a^^rgjqa§3t for "inrome docam'entatioh* Therefore it is reasonable to expect that a 
^ fedgrai reguMory^ requires documentation y/ould likely experience 

' •■•...^ ■■■ ■ ^--flv--^--'' ■ ■ '-^-''-v./- 




ajf high or higher npngomn/lanoB .ra^ep If mlt aQOpmpanleql by even more i^xfcenBive 
enforodment, training, tflminloal aaalafianQe, and rnQnltorlng aoUvltlaat 

• Eighty percent of hpuaehpldr aubjeotecj to the reqCilrement atated they- had no 
difficulty obtaining .the appcoprlata doaumelnt that Qonfrbmed Ihalr eligibility to 
receive free Qp reduoesl»prloe«8cho,ql meala. Only fpur peroenj; i^f the houaeholdo 
Qerected fcr'the' foUQw-ujs* dooumentlon raqueaf reported a-problem providing the , 
document on tlme4A However, informai Intervlewa wlt'h aahWol offlolala auggeat 
.Btrongly that raolplenta only compiled with the -verification requirement , Vj/hen 
conffonted^ wlth the reality that ttlMr chlldren'a free-^^raduced-prlce meala would 
come to an end. Thbrefore theae f lgur<!|p' ma/ uhdereatli^iata the difficulty r^eclplent 
hduaeholda experienced In complylng^wlth the requlremfliW'. ' ! 

^When Delected at random, '5.4 pifrcBnt of ,follow-up\documentBtlon' requlrem'enta 
^conducted revealed. ai^j exceaa benefit awart^. When .reclplenta were aeleoted for ^ 
.verification using the error-prone profile, -the discovered axcess-berieflt rata rooa to 
16 percent. . • . • • " 

Each foUdw-up documentation" requirement took school officials an average of • 
-2i-mlnutB8-to~Gonduct. — -The-average—follow^up-documentatlon-reqUlrement-Gost- 
' SFAs $25.^6 to conduct. This translates Into $478.99 'pen discovered error: using 
random selection and $157.71 ublng EPP selection. The average discovered error 

* save? the federal government $88.31 In meal -relmburaements. therefore for ey^ry 
dollar -spent by- school districts, on follow-up documentation requlremehts, the 
federal government saved $0.18, wheh random selefctlon was used and $0!;56 when 

• EpP selection was used. . ' > , . 

) An estimated 8.3 percent, ot qll applicants subjected ^to_thTB follow-^up 
documeQtatlan requirement did not.cooperat.e and were sub^quently' rorrVpyed from 
the program. '* If the procedure operated effectively,' then this group 
clDmpriBed largely of inellgibles. * Hbwever, mdst ^qf^ tfrose re 
jnoncooperation (7.1% put of 8.3%) were actually' eligible' for program-belief its. This 
means that for ever^ ineligible household retnoved from the prdgram for 
noncooperation with the procedure, nearly six eligible households were removed for 
noncooperation. Hpwever, since more than 90 percent' cOmplie<| with the request,, 
overall only 7.1 percent of the applicants subjected to this'' procedure were eligible 
and were also barred' from participation because they did not ccinnply with' th^ 
documentation request. , > . . 

Findings in sbhool districts not using the IVPP application differed from those 
using the IVPP application in two irnportaht aspects. First, the discovered excWs 



5hould be 
ve'd for 



benefit, ratea wsri mutih higher. When ir^nclpm aeleotlon was uaadi ^ paroanr. 
. error rata .was found, Whyn EPR aajaotlpk wa'^i usadi a 59 parcjenlf' arr'or rata was 
found, .§eopnd| th'a tlma naoassary to Qph^Moi; tha prQq,adure and tha ppst.^par 
' verlflQailon ware muoh lowarr This was tjiia ip two faptora.;, Tha ovarall arrpr rata 
. 'was. mnoh higher In aohaol dlsiirlots nW wWng the IVPP appllQatloni , Also,' the 
majority Qf aphoQl cllatrlotai. not ualng, tJie •IVPP applloatlnn q^^^ th^ 
* ve'rlflafltlQm ti«ing qentr&l SFA ot&ff, whar^oa al^pRt all the aoliool dlatrlata using 
J tha IVPP applloatlon oonduntad tha varlflaetlonn 6n tha Indlvldtml «Qhnnl l^v^U 
, Centralization af the varlfloatloa procona paaultadXln wbotahtlal aaonomlan of • 
,ocQle, j Waga ooata wora muoh lower beoauoa olerloal utaff rather than- achodl 
, prlnolpnla conducted- the procQduro- Training and otart-up cootp ware Ipwar^bepauoa 
.fevvfcar Individuals ware Involved. Time per verlfloatloa waaXlower beoauaa of great^ 
routlnlzatlon of action. ^ \ 



\ 



Quality: AaBuranfce PriOcedure V: Parent^ T^elephpne Conference U ^ 

the results of the parent telephone conference are preoent^ed by cdntraatlng 
. . school diotrlcts thajt ijsed the^^^^ with school distrlctij.that dfd not uoe 

the IVPP' application.,) ' - \ * 

Forty-two, percent . of ceciplenta selected for parent telephon^ conferences, 
reported hevef having been contacted. Afs with the" follow-up dc^c^mBntatlon 
requirement!, this finding suggests that without extensive enforcement, training, 
technical assistance, and monitoring, a federal regulatory requirement Njof parent 

conference verlfloation would also experience a high noncompliance rate. ^ 

■ - ■ ■ • ' " ■ . . . ' ■• ■ ' ' ■ , ' ; ' ^ , , ' ^ ■ " \ ■ ■ 

.Of the recipients fcontacted for parent telephone conferences, less than 1 

> 'percent refused to cooperate. The average parent confai^ence requir9d' ?3 minutes, 

. * including ' time for call-backs arid outcome . report completion. oThe average 

• conference cost for school districts was $13,81. * 

usingV raf^dom' selection, parent ^conferences revealed a 6.1 percent 
^ excess-benefit rate. Under EPP selection, the dlsqovered excess-benefit rate rose» 
to 13 percent. Given average sayings to the federal government of $88.31 per 
discovered error, it cost locsir school districts $1.00 to save the federal government 
$0.39 in excess-benefit payment^ when random selection was used and $0.83. when 

; . , ■ ■■ / , • 

' EPP selection was used. 



. Jim typerof Ehlrcj''P8rly varlflQBfcion prpoatliLjpis were Qoncluptedi' vsrlflssllnm 
^through iqoal wilfare QfflQSfl ancj.wage l^ippme mstohei with' Qompi-it§pli§fl 
wap • fll§8*- BeQauae iuQlv third-party. Qh80l<8 raqulred elgned ^AhorUationa by ' 
ap^lloflita,^thiiy aould only b§ QOrtflygt^d In- aohoQl fllstrlQta that uaad tha WPP 
appiloa^Uon bapauee It InaUidad' suiih an authorisation! The kinal walfara 
veplflQatlonB wera qonduetad oantrally by' SFA- staff baQausa of tha naad- to 
cjdopcllnata obntaot with looal welfapa offlqas. Tha atata wage tapa" matoh waa 
oonduatQd by tho evaluation Qontruatar, In ana atata for four pilot ^ahgol dlrttirlota, 
Thareforo tho wage topa match raoulta muot ba aaan a naoa latudy and «hmild not 
ba ganarallzed. Furthari baaauea ^tha fivaluatlon qontraotor aonduatad tha' waga 
motoh, ooflt flgurao ara not.'avallablai Aolda from «at-up, thaaa nbat^ uhouki 

almllar to coata for local loval third- part/ matohaoii ^ , , / 

• . ' ^ ' ,, , ^ • ^ « i , • 

Locar third-party welfara^aheckB had a dlocovared 'axcoQ«i«*banbflt rata, of laoo 
than 1 percent* In half the ochopl dlatrlcto uslng'thlo procodura, not a alngla error 
waa diocovared. Thlo reault .was expected because local welfare offlcea 4^avo • 
information only on individuals that participate In the food stamp or welfare; 
programs, and priori analysis by IVPP showed such Individuals to have a very low rate • 
^ of excess benefits. 

The state waga tape match was much more successful with a discovered excess 
benefit rate of U.6 percent under random selection and 35.2 percent when EPP 

selection Was used. 

■ • • ■ ... ^ ■ 

' The findings collectively suggest that quality assurance procedures can be 
implemented in the "free and reduced-price school meal programs and that it Is 
.possible to reduce misreporting. Phase II of the Income Verification Pilot Project 
; suggests that the most successful quality a^sur-ande strategy relies primarily dn 
deterrence. A carefully designed meal benefit application produced the greatest 




pedMQilfln In m^t ^IUh'ths Imi Qmt m(\ MwmimQM }.q tesholfis and sphflol 
dIalrlQtSi VeirlflRsllgn qf rsglpleriis iftap thiy Wire approved He) peqeivii m.n\ 
bQneflls wa§ most; 9ffi«Uv§ly ^peofnpllihpd by flr^t aelaeting sppllg^ilpn^ wUh ,a 
high Hks'llhnod.af irror using an 8Prpr« prona prQflls. Th§ verlfleifelnn mefehQd ysid 
myst be able to deteot aarnid lnQomii\(3f the mgthPda tesl-ad by, IVPP| thlpd-papty 
rHbrN with agenoiea that maintain w«g§ and aarnlnga raonrda shnwid tha mbat 
promise* \ ^ V , v 

Q.A eONCI^JSlONS ^' ' . , ^ 

Phaaa II of tha Inooma VarlflQatlnn Pilot Prtjlaflt waa a natU^nal axpaHmaht that 
toated 88ven quality aBHiirfjnoa pj'oaadiirafi daalqnad td ratliiaii applUiant arrop In tha 
aohool ^maal programi. Thaaej lavan prooadupii 'were davalopad by adapting 
prooodurftu uaad In otl^ar fad^ral-prngrama and In eoma aohnol dlstrlatH. Oaoauaa fcha 
p«r-raQlpl«nt benQfltfl bvala ara low ($2QQ par cljlld par yaar), thatftk adaptations " 
fooufied on minimizing tltti,^ and affort to Implement tha prooaduraf}. THa navan 
procoduroo conalatad o?- two d6t8,rrflnt prooedurea that wora axpaotod^to pravont 
errora before boneflta Were owordad and fiva dataotlon proopdurna that warn 
expeotad to deteqt arroro aftar bone Tito wara iiwardad. Cone lUolono about all of the 
proGoduroo oro oummorlzod In Exhibit Q.2 nnd dlocunoed bolow. Ther oxhlblt 
doacrlboo a variety, of quality paouronco Implamantatlon ond-effectlvonooa faotora 

rejatrvo to each of the prooedurea tested by tho IVPP. • • • 

. . . ■ ■ ' ■ , . ■' " . ■ ✓ ■ 

CONCLUSIQN 1: THE REVISED IVPP" APPLICATigN EFFECTIVELY REDUCES 
• MISREPORTING 

. IVPP Appllcatlonfflubatantlally RBducea Program Error Rate . , 

, The revlaed IVPP application, which . requeato detailed Income 
reporting '-and Informa appUcanta. that they may bo aubjacted to 
verification, reduces the program , error rate' to 11.7 .percent, oa 
' compared with 23 percent for non-IVPP . appllcatlOna. The 
application virtually' prohlblta Inellglbles from receiving beneffts, 
with only 3 'percent, of all 'benefits diatrfbuted to Inellglbles. 
Further reductions in this error category would be very difficult to 
* achieve. ■" • 

IVPP Appllcallon Reduces Error Dolldr Uosaea. 

' When the lU percent error rate is translated into error coats (i.e., 
' V. ^ . the percentage of dojlafs lost to fraud and abuse as a function of 
total program expenditures), the dollar error rate is low, 
approximately 5 percent. Relative to other procedures tested, the 
IVPP applicatioh produces the greatest savings because it prevents 
the misallocation^of meal benefits for the entire school year and it 
affects ail, not a sample of applicants. 

•87 ■ . 
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> 1/ Hofori only to'liicroniiiiitft). flffficts of ilflciiinontatlon, m\) tlioii(|lr«|)|)lk8tlflii 1* nocoiiary iiKl jIjo \m effocti. ' 
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'EXHIBIT B.2: SUfW\HV OF CONauSION$(ConUnued) 



: ■ # 



PLITY ASSURANCE PROCEDURES 



111 ' ■ 
APPLICATION 

' - • PLUS -FOLLOH-UP , ' 

\ 'II DOCUHNTATION OOCUftNTATION ,Vl mi 

, APPLICATION^ PLUS, PLUS V ' LOCAL ♦STATE 

I , ^. PLUS ^ CONSISTENCY CONSISTENCY ^PARENT' WELFARE 'WAGE 

APPLICATION DOCUMENTATION 1/ CHECK 2/ CHECK CONFERENCE CHECK MATCH 



Cost/Burden to Applicants 
Cost/Burden to Third Party 
Barrier to Partlclpjtlon 
Desirable Scliool Configuration 



Amount of Assistance 'to 
School Districts Needed to 
Assure Success of Procedure 

Recjuires Updates/Hodlflcations 
to Sustain Effectiveness 

Requires Applicant Cooperation 

lequires External Agency 
Cooperation 



Low 
None' 

Low or' None 

Any Configu- 
ration 

Low, or None 

Essential 

Yes ' 
No 



Hediun 
Low or None 
High 



None 

Low or None 
None 



Any Configu- Centralized 
at ion Preferable 



ration 



Low or Medium Hediun 



Niiffl Low llofie None 

Low or NonO' None. Low Low 

High Low ot* None None None 

Centralized Centralized ^Centralized Centralized 

Pt'eferable^.P/eferable Necessary Necessary 

Medium y Medium High High 



Desirable Desirable Desirable Desirable Desirable Desirable 



Yes No 
Occasional Occasional 



Yes Yes 
Occasional No" 



No 
Yes 



No > 
Yes . 



1/ Refers only to incremental effects of documentation, even though appllcatloii is necessary and also has 

2/ Refers only to Incremental effects of consistency clieck, even though application and documentation are 
have effects. - ' V • 



effects. , 

necessary and tliey also 
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There l3 No Evidence to Suggest That the IVPP Application Acts As 

- . . a Barrier. ' > ^ ^ 

.., ■ ■ 

, ' In view of the removal of virtually all inelgibles from the pjrogram, 
the IVPP application was associated with reductions in program 
participation. However, the revised study application does not 
^present a discoverable, barrier to participation for eligible 
households. •^ ^ . ^ 

IVPP Application Is the Least Expensive Procedure for Schbol 
Districts to Implement. 

Although it costs school districts about $1«30 mora to process the^ 
IVPP application than non*IVPP applications, the IVPP application 
was much less expensive for school districts to iroplement than any 
■ ' s . of the other quality assurance procedures tested. Further, this cost 
^• should diminish in subsequent' year^ as both schools and parertts 
become more accustomed^ to the IVPP application's reporting 
requirements and format* 

. ■if'' : ■ ' ■ " ■ •• 

• CONCLUSION 2: SUSTAINED EFFECTIVENESS OF THE IVPP APPLICATION IS 

UNKNOWN I 

9 ' 

, It is believed that the IVPP application reduced unintentional 
— misreporting by clearly indicating the types of income to be 

reporti^d for each adult in the housenbld, and reduced intentional 
misreporting with the explicit threat of possible verification. 
However, because; the pilot study was of limited duration^ it is not 
\ now possible to 'estimate the continuing effectiveness of the' 

application. It Is likely that verification will have to be carried out 
to jome degree to deter intentionat and careless misreporting. 
However, study findings do not offer guidance on how widespread or 
intense income verification procedures need to be in order to sustain 
the deterrent effect of the application over time. ^ 

CONCLUSION 3: DETECTION PROCEDURES ARE NOT COST-EFFECTIVE ^ 

None of the tested income verification detection procedures save 
^ more money than they cost. Moreover, these costs accrue alfnost 
exclusively to school districts and not to states or to the federal 
government. The high local costs to conduct verification are due in 
large part' to the fact that higher paid school personnel (e.g., 
principals) typically conduct verification. Detection procedures do 
not produce substantial savings because they are implemented after 
applications are approved. In contrast to the revised application, 
the detection procedures, in general, present a barrier to 
participation for eligible households. — 

0 
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. Procedures Involving Documentation Are Lalat Feasible . 

bocurnentat|on requested with the appllcgtion and after the 
application/' was approved produced baa^riers, thereby barring 
V eligibles ffoai the program* Document requests were expensive for 
scj[iool districts to process, and the document did not achieve 
substantial error reductions over and above the application* 

' ^ /■ . ' ■ . 
Local- third-Party Welfare Checks Are Ineffective, 4 

Although less expensive for school districts to implement because 
of economies of scale, third-partychecks with welfare offices did 
not detect errors* This is probab^because welfare recipients are 
often eligible for meal' benefits, 
/ ' • • . ■ ' * ■ ' 

;/ ' State Third^Party Wage Matches Appear Effective, But Would Be 
Difficult To Implement* 



{^J, A wage match was conducted by the contractor for school districts 
/ in one state. The match was highly successful in discovering 

errors. However, 4t^is probably not feasible for all school districts 
to conduct wage matches. Matches require\the cooperation of the 
.state employment security agency. Sevepm states do not possess 
the. data and others have legislation andpolicies preventing them 
from participating in wage niatching. It would be very burdensome 
and, therefore unlikely for state agencies to deal independently 
with the numerous school districts in their state. / 



Parent Conferences Are More Cost^Effective Then^'^ Other 
Verification Procedures, But Are Still Highly Ineffective. 

Of all verification procedures tested, parent conferernces were 
most cost-effective* However, the parent, conferences were often 
not implemented properly and did not identify very many instances 
of misreporting. 

Use of Error^Prone- Profile Makes Detection Procedures More 
Effective. 

Targeting applications foi^ verification using the erj^'r-prone profile 
is more effective in locating errors than random selection. The 
most effective detection procedures werei^ parent conferences and 
state wage matches using the error-prone profile. However, even 
using the error^rone profile, parent conferences were not cost- 
effective, at leastin the short run. 



CONCLUSION 4: IMPLEMENTATION PROBLEMS ARE ENCOUNTERED BY 

Sp(400L DISTRICTS 

In pilot sites, even with extensive evaluation contractor training 
. and technical assistance, school districts failed to follow through 
with full implementation of several detection procedures. Such 
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compllanae problems may be encountered under normal operating . 
conditions/ Study finding3;^suggest/t this is partly due to the - 
complexity 'of the quality assuraj^e procedures, requisite follbwup 
^activities, and built-in administrative disincentives- to correctly 
implement qualtty assurance procedures. 

CONCLUSIONS: ' COSTS AND OTHER FACTORS ACT AS STRONG 

DISINCENTIVES FOR SCHOOL DISTRICTS TO IMPLEMENT 
QUALITY ASSURANCE PROCEDURES 

' * • " ■ '• ^ ■ • 

. * As implemented in the pilot study, all costs for quality, assurance 
were borne by school districts, local and state agencies, and by 
parents, yet all savings 'are accrued to the federal government.fv 
Further, school districts are typically not structured to carry out 
quality assurance procedures efficiently and they sometimes do not 
view it as a role appropriate for educators. 

Taken together, these results suggest thptV deterrence procedures, and in 
particular an improved application such as the IVPP application, would result in 
substantial cost, savings with the least disturbance to schools a|d recipients. 
Because the IVPP applicatibg is so effective In reducing costly errors associated 
with ineligibles receiving benefits (versus elig|bles receiving free meals instead of 
discounted meals), subsequent detection procedures will not be very cost effective. 
It must be kept l^pind, however, that the p/PP application was tested in only one 
school year, and available on its long-term effectiveness. 

In addition to providing more direction to applicants on how to correctly report 
Income, the IVPP .application clearly Indicated that the application Information 
would be subject to verification. WitHtiut subsequent verification and publicity of 
the results, It Is likely that th^ deterrent effect of the application would diminish. 
To maintain the deterrent effect of the application, the logical next step would be 
to implement a limited verification program.- Based on study findings, jt appears 
that the use of the error-prone model in conjunction with state wage matching 
would be the most cost-effective tool. State wage matching was carried oUt by the 
evaluation contractor and. the study does not provide guidance on feasibility of 
implementation by school districts. However, there would be costs to the school 
districts apd state agencies^volved. Clearly, careful planning and further testing 
would be v^arranted before .such a program is implemented. Based on study findings, ^ 
parent conference would be the second choice for a verification method^ if wage 
matches are not feasible. , . 
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END NOTES 



V; . ^ ...» 

1/ .1 



Households determined through verificatidn to ; be ineligible for free or 
reduced-price school meals^jan. elect ,rt>-pay full price for school meals, JThus 
when households elect to pay full price, the school district will c6ntinue to 
.^receive income. However, when the ineligible households completely withdraw 
from participation in the school meal programs, the school district will suffer 
an income reduction that makes it more difficult to*mefet fixed program costs. 

■ . ' ■ ■ ' . ' ' 

2/ In thre^Jajrge school districi^s that did not use the IVPPapplicatlon,^a 5 percent 
^ samfile ot^approved applications was used. Although the design includes control 
^ . sites, v6^^i^at ion vyas notronducted ia these school districts. ^ ' ^ 

3/ The remaning 16 percent of participants subjected to the dijcumentatiori 
following application with consistencv check were unable to describe single 
specific problems. V ' 




I ■ ' ^ C' 
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V APPENOIXA ^ 

IVPP APPLICATION AND INSTRUCTIONS 
■ .TO APPLICANTS REGARDING THE 
' PROVISION OF DOCUMENTATION ' 




i 




' ( 




^14 



REDUCED SIZE-NOT 



SIZE: 



'i -skill -^-y^: : .■ -. •'■^^ - V/;-: '--^ 'V ■ 

lAPPLlGAMONyOHiPRE^^^^ SCHOOL MEALS-^SCHOOL YEAR 1982-83 

^^.iT •PFOIcjtloo f«rEAOT sqboql childrw atMoci A compUu appUoiUoa must htV6 ehe names ol aU hoLhoia 
:/ni«ib« tha ^ S^urity Nmnbani o( dl bouMbold m«nb«ra 21 y«ara wJoWar. aU houaahoM iocoma. and tha appUcaUob niuit bo^miS by « 
d mamoar such aa a oaraot Contact your school if you hava fostar^dran and »<.ti f» m^\^ r»« di^.vI oDTxm« v^*.*^^ u 



, Ihii^Li ^Z-l^uT" •« ^"/-^"".M ^-^^ wjw-, au nouaeiuMd locoma. aoa tna appUcaUoo muit ba ainied by ao adult 

^^S^^^^^ "T^ " • parant Contact XOt^ «hool if you haya foatarJhUdran |u^d wtsh to apply for tham. Pleasa PRINT neatly, Joint L or 

. pjncH and praaa flrmly-^yoa are making copiaa. Return ALL compiacad co{r[aa to the school »/. wo- . j^uii, h«u w 



NAMEOPTHESCH<X)tTHECHILDIlENAVrEND:. 



/ ^FULL NAME^^ GRADE OF EACH CHILD ATTENDING THE SCH^L YOU PRINTED IN NUMBER 1: 

/ _ , . . ^• 

LAST NAME FIRST NAME 

(O '_ »• ■ 

tD> ^ . .. 



.'^liAsTNAMB 



'mm 



FIRST NAME 

'-Kr— — 



GRADE 
IN SCHOOL 



GRADE ' 
IN SCHOOL 



W0'?^^^S^ TIffi ADULTS IN YOUR HOUSEHOLD, print tba foU name in column A of ALL raUtad and'unreiatad parsons UviMli ycnlr 

d^^?^mJ^^Th!^Jl^ 21 yaara or older. W an adult doaa not have a Social S^rurity Number, p^t tba word NONE In the epaa, ^ ' 

:£ifi^ ■ '^^l^^^^^^^!^^ ojjf^wi. e«ch parson ™ ACTUALLY PAID OR RECEnWLASt MONTJI from eai, of ^he eourcee ebown beloW. If a ^ 
jwsonracaivad mora than one cbwJt or caab payment «ro^ . ^^'^ ' 

the parson wm under the en of 15. nrint tha wnnf monk in .^n^^w.*. r« i *^"*:r 




II. J L — '^--.^ w, uvu« » WWW snuw uw amouDv II e person 

STd????!.'^ J?* pS** ^ ^ ''^''^ the approprieu apace; If a person'e inconu Uet month 

CIRLUS tba eetimatad avtraga monthly amoun^tbat parson expacta to raceiviB from that aourca during tba year, ' 



Last and First 
Naipa'of • 

Household 
Member' 

><A) 



Aga 

(B) 



Sodal 
Security 
Number 

(O 



Earnings 

From Job<8) 
iBefora 

Withholdlss) 



"7" 



Welfara 
Paymenta . 

iAFDO 



Social : • C 
SecuriC:^ SuppOi 
(Green or ' and/or 
Gold Ch^kl.^ Alimony 




lya income firom a source or 
fypical you muat print and 



All Other 
-J Income 
^iSee Note) 



t . ■ 




1 


t 


a. ■ 


- 




. .. J ' 




_ » 












1 '_ 






« 


1 


s 




1 


a 


s _ 


1 


1 


_ 1 



TOTAL MOrsPTHLY If OUSEHOLD I|^COME S , 



NOTE: a jOther income mdudeapenaionaand'retirement^^^^ J ' 

m\oM do^not have to include income received by children for occasiooal or casual jobe such aa lawn work or baby-sitting. " ' " 
13) If you need to compute e montldy income anwunt, sea instructions on the back of the eppl^ \. . 

4 CHECK THE APPROPRIATE BOX: lyour anawerjs voluntary. Houmfa^ that receive^ stampa-can also t^ve school 'mealaJ ' ' 
' . □ My household does' not currently receive food stampf. □ My houaah3acurrentiy receives food stamps. ' 

JlXlvf ^ **^£ *^ hiformatipn is tnie and correct and aU houLboId Income has bmn reported: understand that deUberate misri^prasentation of 
^l^^n^^'^^^S^^^? nay subject me/us to {^oaacution und« apkicable suta and federal sUtutes: understand.that this information is^g given 
in connection with the receipt of federal fimda; and understand that schod offldaU may verify the information on the epplicatioo. " 

□ ^^^l^X^"^^ P^^iST contained in Food Stamp Agency. Welfare Agency, or 3Ute Employment Agency fUe« may ba 

L-J uaed to verify the information provided on thia application for free or reduced price school meal benefits^ ' ' ' - 



. SIGNATURE OF' ADULT ! ' . • 

^HOUSE^HOLD MEMBER SIGNATURE OF OTHER 

WHO COMPLETED APPLICATION ADULT'HbUSEHOLD MEMBER 



SIGNATURE OF OTHER 
ADULT HOUSEHOLD MEMBER 




. DATE; . 



. DATE; . 



• SIGNATURE OF OTHER • 
ADULT HOUSEHOLD MEMBER 



, DATE: 



HOME ADDRESS. AND TELEPHONE NUMBER OP THE ADULT HOUSEHO^J) MEMBER WHO COMPLETED THE APPUCATION. 

[EADDRESfl! » , ^ '"^J ^ " 

— — — ' — — 

DAYTIME TELEPHONE: ■ ' ■ ■ ; 



I 




%' . ' FOR SCHOOL USE ONLY 

. A. TOTAL MONTHLY HOUSEHo£d INCOME: (From «a) % ' ' • 

TOTAL HOUSEHOLD Sizi: ijotal Number of Nimea From#3?-« • 
C. APPROVALSTATUS: ^ q^pproved Free lAala 
a DATE PARENTS NOTIFIED: ____DEri7:RMimN00FPl<JlAL: _ 
E. REASON FOR DENIAL: 



a Approved Reduced Price Meela 

— :■ — ■ — — I — 



: Denied (Explain Belowl 
_DATE:-^ 
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■ J DOCUMENT PROVING HOUSEhbLD ELIGIBILITY - 

application to be-considered complete. YOU ABE'reQUIRED TO SXJBMIT A COPY 
OF AT .L^STJONE OFFICIAL DOCUMENT that confirms pr nrSv^s'the eUgibiHty of yoi hi^S ' ' ' 
to receive free^or reduced pnce meal benefits pie instActions befc t»U you exactly what document you • 

/Cist' ??KM P'^J^.'^^r^^ Qorifedential and willOje used to assist in proving your . 

house hold^ ehgibihty for f^ee or reduced price meal BbnefitsU j ^ T • , 



1.^ Y<^ JIQUSEHOLP RECEIVE^ 

documents. If you dp not w^t to do this/ dp 2 or i^l p^^^ '^^ ' . . 

• Food Stamp Certifitation Notice s^ne to^odi y ^h^ Food &amp Office, Oft . 

' ■' ' ' ' ' I ■' ■ ■ ■'° J^'M' • ■^•l-^'!' . ■ ' ■ ' ■ 

• A letter -from the Fo6d' Stamp Office |ji|Jinjp tg 



2. IF Yq(^P .NOT DO #|ANplffi|MpjR H0USEH0I4) ARE EMPLOYED: You must sub^ 
nutoneof the f6UQwinff Xmirt&.fef50 / 'Hfov """^ 



mit onQ of >^he foUowihg person. 



• ;Pa3(env^lope jshov^gr ydmfMal'^bs^^ bR 
, • A lettertf oA the empibye^i^^^^ your total gfqss wages, OR 



^* f«lf;f'PPl7^ P«raons W copv^^la receipt sales tax statement, or PICA form, or.last 

. . qufu1»rly tax estunatpMf ; 'V ^ , » , ' ' : - . 



OR 



3. IF YOU DO 

You must 



10 NOT DO #1^R #2, A]jn) NO ADULTS IN'YOUft HOUSEHOLD ARE EMPLOYED: ' ' 
submit a copy of the document for the largest; source of household income. • 7 ' 



IF THE LARGEST SOURCE OF HOUSEHOLD 
INCOME ISr ' 

Public assistance (welfardrAFDC) 

Ufiemplbyment compensation ' ^ ' 

, ■ - :■ ^ 7 : • 

Social Security , . ' V ^ ; . 

Alimony pr Child Support 

Disability ot Workman's Cohipensation 

v^Veteran's Benefit's \^ - ^ > V 



VOU MUST PRdVroE: 

Your benefit notice from th^ welfare, a^cy^^ ; 

Your notice of eligibility^ ^om the State - 
Employment Security Office - ' . 



Your SSI eligibility letter or your Social sedirity 
Retirement benefit letter ^ . . . ^ * 

A cop^ of the court decree or agreement * 

A, copy of the disability award letter, or chl«k ? 
stub 

A Qopy of the benefit^bilce from the'VA " 



OR 



^Tl^^^^OTDOi^ 

You must attach^a brief note explaining in writiig hoW your house];i61d pays for food, clothing, and 
■ , / . housing, and whejv^ou expect to have income. 
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Submit Your Doctiment With Your Applitation 



He 



i3e 



SCHOOL DI^TltiaWCTERISTIC 



SFA 1 SFA 2 SFA.3' ■SfA:H.SFA 5 SFA 6i SFA 7. 



FNS Region 



7 



West Uest Uest Nld/Atl Hld/Atl Hld/Atl HId/AtI ' 
"fiOO . 5,01?, 15,000 . 5,2'27 . 8,200: 10,588 17,390 



Percent'of Students Receiving Heal Benefits ' 36^^ ;^^ ' 1^ '35^' 15.2 22.6 21.9 

3 9 . 25 ' 21 ^ '19 , /« 49 . 



Nimberof School Buildings ; 
Uhere Applications Here Frocessed 



School .Central Schoor School School School School 
Building (SFA) Building Building fiiilldlng Building Building^ 



Hhere Quality Assurance Procedures Here • Schdol Central School School^ School School Schobl; 
Conducted Building (SFA) : ilullding flulldiri^' Building Building Building 



NinberoflVFP .Applications Processed 
• V Total Nifflber'of lVPP Applications (0^^ 216 3,021 

,/ lVPP Application Only-' 

, (puallty Assurance Procedure I) t» 

IVPP,.Appllcation HIth Docuntentatlon ■ 
(Oiialtty Assurance Procedure II) 

Hudiber of.Doctlment Consistency Checks , 
Performed (puallty Assurance • 
Procedure III) 

. Number of Ddcument After Application 

With Docunient Consistency Check Performed 
, (Huallt^ Assurance Procedure IV) 

■ y • 

Number of Parent Telephone Conferences 
Performed (Quality Assurance Procedure V) 



(Quality AssijranceProccdur^e vjj 

umber of State ThIrd Party Checks P 
(Quality Assurance Procedure VII) 



CIO 


1 1171 


m > 


(9) 


216 ' 


1,632 


(NA) 


; (4) 


NA 


1.389 


(NA) 


. (51 


NA 


NA; 


(NA)/ 


(NA) 


,1 . 
(3)- 


(NA\: 


,. i/ 




NA 


. 83 ' 


(NA) 


(?) 




, 81 ' 


NA , 


!1 


(NA), 





576. 



{in 2,162 21761 

D-st (19) («) ; ■(«) 




57 



(13) (26) (33) 

935 1,5B2 
(6) (17) (16) 



28 ' ■ ' « 
(9) (7) 



37 47. ^78 
(6) (13) (16) 



66 113 



(1,0) m (21)' . (26) 



129 ' 
(21), 



149 255 
(43) (49) 



„ 129 105 149 . ; 255 
(9) Hi) ^ .(21) (19) !i (43) (49) 



(1) Nuiers shown In parentheses represent the nimberof schools 1n«oli/ed In each Quality Assurance Procedure. 
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• SELECTED, CHARACTERISIICS OF EXPERIMENTAL SCHOOL OISTRiaS 



r 



■ : SCHOOL^OISTRICT CHARACTEfllST ' ' ' 


■ SFA 8(2) SFA 9 


SFA 10 


SFA 11 


SFAir 


SFA 13 


FNS'ieglon 


South/' 
East 


South/ 
East 


South/ 
East 


South/ 
East 


South/ 
East. 


Hest \ 


Enrollment ; . » * 


22,000 


3,959' 


11,244 


22,770 


7,212 


6,600 


Percent of Students Receding Heal Benefits 


26.0 


47.9 


54.1 


,33.4 


34.7 


60.3 


Number of School Buildings 


: 38 


12 


20 


. 44 


13 


16 


Where Applications Were Processed 


School School School' 
Building Building Building,; 


Central 'School Sctiool 
(SFA) Building Building 


Where Quality Assurance Procedures Were ^ 
Processed -:i3p 




School School School School 
Building Building Suliding'r Building Bu1.1ding 


Number of IVPP Applications Processed 


. t '■ , _ 
I ( ■ 












. Total Number of IVPP Applications 


5,950 
•(38) 


2,510 
(12) 


7,814 
(20) 


10,608 
,(44) 


3,109 
(13) 


5,366 
(16) 


■ IVPP Application Only" ' 
■ (Quality Assurance Procedure I) ; 


5,950 
(38) 


1,579 
(5) 


4,782 
(14) ^ 


, 7,100 
(29) 


1,776 
(9) 


2,002 


IVPP Application Hitli Documentation' 
(Quality Assurance Procedure II) 

• . ■■■■■ ■ J 
Nmnber of Document Consistency Checks 

Performed (QOallty Assurance 
- Procedure III) 

'/'^ • ; ■ ■ ■ 


NA 
(NA) 


, 961* 
(7) 


3,032 ' 
(6) 


3,508 
(15) 


1,333 
(4) 


'3,364 
(6) 


NA , 
(NA) 


51 
.(4) 


57 
(3) 


' 61 
(7) 


26 ' 
(2) 


- 34 
(3) 


Number of Document After Application 
V With Document Consistency Checic Performed 
(Quality A'ssurancfe Procedure IV) * 


HA 
(NA) 


NA 
(NA) 


120 
(?) 

■ * 


176' 
(14) 


61 

.,('5) 


79 
(4) 


Number, of Parent Telephone Conferences 
Performed (Quality Assurance Procedure V) 


■NA 
(NA) 


■ v65 
(8) 


' 172 
(10) 


240 
(23) 


65 

(6) • 


112 
(9) 


Ndmber of Local Third Party Checlcs Performed 
jQuallty Assurance Procediire VI). ; ^'j?; 


NA 


■ 126 ' 


385 
(20) 


497 
' (44) 


157 
(13) 


212 
(16) 


Number of State Third Party 'Checics Performed 
, (Quality Assurance Procedure VII) 


' ' NA 
(NA) 


176; 
) (12) 


522 
(20) 


■ 736 ■ 
(44) 


249' 
(13) 


0 

(NA) 



(•* 6-*)ers shown in parentheses, represent the number of schools Involved in each Quality Assurance Procedure 
(aA^J school district was iable to fulfill the Qua^^ 



SELECTED CHARACTERISTICS OF EXWRIMENTAL SCHC|OL OIST 



SCHOOL DISTRICT CHARACTERISTIC 



FHS Region 
;Enro11ment. ' 

Percent of Students Receiving Meal, Benefits 
Number of School Buildings 
Where Applications Were Processed 



Ifhere Quality Assurance Procedures Mere 
.P/recessed 

NuTitier of IVPP Applications Processed . 
Total Number of IVPP Applications 

\ IVPP Application Only 
^Quality Assurance Procedure I) 

^VPP Application, With Documentation 
(Quality Assurance Procedure II), 

Number of\ Document Consistency Checks 
Pprfonned (Quality Assurance 
Procedure III) 

Hurober of Document After Application 
. .With Document Consistency Check Performed 
(Quality Assurance Procedure IV) 

Number of Parent Telephone Conferences 
Performed (Quality Assurance ProceftJure V) 

Number. of Local Third Party Checks- Performed 
(Quality Assurance Procedure y I); ' ^ , 

Number of State'Third Party Checks. Performed 
(Quality Assurance Procedure VII) 



SFA 14 


SFA 15 


V SFA 16 


SFA 17 


SFA 18 ' SFA 19 


South 


North 


North 


North 


Mid 


Mid 


West 


East 


East 


East 


West 


' West 


5,700 


1,649 


2,686 


6,305 


1,336 


2,859 


. ii.o 


31.2 


■ 30.0 


2.2 


20.8 4.6 


5 


- 5 


■■■ ■■ ^ 


8 • 


5 


5 


Central 


School 


Schol 


School 


- 

Certtral 


School 


(SFA^) Building 


Building 1 


DUllding 


SFA) 


Building v 


Central 


Central 


School 


a 


Central 


School 


(SFA) 


(SFA) 


Building 


(SFA) 


(SFA) 


Buijldlng , 


558 


1.151 


940 


• r 

279 


308 


135 


(5) 


(5) 


(6) 


' (8) 


(5) 


(5) 


NA 


NA 


nil A 

940 


279 


NA 


135 


(NA) 


(NA) 


(6)' 


(8) 


(NA) 


(5) 


558 


1,151 


NA 


NA ^ 


V 308 


NA 


(5)' 


(5) 


(NA) 


(NA)^ 






7 


15 


NA 


NA 


22 


NA . 


(3) 


(3) 


(NA) 


(NA) 


(3) 


(NA), 


15 


NA 


NA 


' 6 


NA 


(13) 


(3) 


(NA) 


(NA) 


(4) 


(NA) 


(51 


11 ' 


8 


51 


5 


NA 


NA 


(2) 


(2) 


(6) 


(4) 


(NA) 


• (NA) 


"NA 


NA 


56 


NA 


11 


13 


. (NA) • 


(NA) 


(6) 


(NA) 


(5) 


(5) 


NA 


NA 


NA 


NA 


11 


.13 


- (NA) 


(NA) 


(NA) 


(NA) 


(5) 


(5) 



(1) Nuinbers shown; In parentheses represent the number of schools involved In each Quality Assurance Procedure 
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SELECTED CHAIWCTERISTICS OF EXPERIHENTAL SCHOOL DISTRICTS 



V SCHOOL DISTRICT CHARACTERISTIC 


. ^^FA 20 


SFA 21 


4 

SfA22 


SFA 23 


>A24 


SFA 25 


FNS Region 


Mid 


South 


1 

South . 


South ' 


South 


Hid 




'J West 


Nest 


Hest 


Hest 


Hest 


Hest 


FnmllniAfi^ ' V 


ooU 




767 


8,029 


3,813 


40,000 


Percent of Students Receiving Meal Benefits 


16.5 


42.9 


26.7 


21.6 


3Z.9 


25.6 


Number ^fSchoolBulldings 


2 


/16 


4 


15 , 


. 6 


60 


Nnere i^ppiicaCions Mere rrocesseo 


School 


School 


School 


School 


School 


Central 




Building Building Building Building Building . 


(SFA) 


Where Quality Assurance Procedures Werei 


Central 


School 


•School 


School 


School ; 


■Central 


'Processed 


(SFA) .Building Building Building Building, 


(SFA) 


Number of IVPP Applications Processed 




• ■ ■ ■ ' 










locfli nuniDer or ivrr Applications 


NA. 


3,045 


NA 


1,664 


1,318 


kit 

NA : 


(NA) 


(16) 


.(NA) 


(15) 


(6) 


(NA) V 


. ivrr 8ppiicat ton Only 


NA 


2,189 


' NA 


1,190" 


1,318 


NA 


(Quality Assuranof Procedure I) 


(NA) 


(11) 


(NA) 


(9) 


■ (6) 


(NA) 


IVPP Application With Documentation 


NA , ' 


856 


NA .' 


474 ■ 


• NA 


NA 


(Quality Assurance Procedure II) 


(NA) 


' (5) 


(NA) 


(6) 


(NA) 


(NA) , 


Number of Document Consistency Checlcs 


NA 


24 


10 


10 


' NA 


' NA . 


Performed (Quality 'Assurance . 


(NA) 


(2) 


(3) 


-(2) 


. (NA) 


(fA) V 


Pi^AraHiii^o ITT\ 
rfOttSUUrtf 111^ > 




Number of Document After Application : 


NA 


85 


NA ■ 


31 ' 


NA 


230 


Nun'uocumenc tonsistency inecK t'erTormeci 


(NA) 


.(6) 


(NA) 


(^) 


(NA) 


(19) , 


(Quality Assurance Procedure IV) 






nuiiiiier or rarGnc, leiepnonG loniGrcncGS 


. 15 


95 


-NA 


55 


102 


219 


Performed (Quality Assurance Procedure V) 


(2) 


• (fl) 


(NA). 


•(9) 


' . .(6) 


r (20) 


Number of Local Third Party Checks Performed 


NA 


222 


12 


NA 


102, 


246 


(Quality Assurance Procedure VI) - 


(NA) 


(16) 


(4) 


' (NA) 


(6) 


(21) . 


Nunber of State TJiird Party Checl(s Performed 


15 


NA 


NA 


• 'na 


NA 


V NA 


(Quality Assurance Procedure VII) 




(NA) 


(NA) 


, (NA) 


:(NA) 


..(NA) 



(1) Numbers shown In parentheses represent the number of schools, Involved 1(i each Quality Assurance Procedure. 



SELEOTED XHARACT£RISTICS OF EXPERIMENTAL SCHOOL DISTRICTS 



SCHOOL DISTRKJT CHARACTERISTIC 


SFA 26 


SFA 27 


SFA 28 


SFA 29 


Total 


FNS Region'^ ^ . 


•4 

. Mid 
West 


Mid 
. West 


.^.Mid 
West 


Hid 
' West 




Enrol Iment * • 


51.0B3 


70.516 


7,128 


. 4l;775 


379.463 


Percent of Students Receiving Meal Benefits 


\36.3 


47.7 


17.4 


45.9 




Number of/ School Buildings v 


105 . 


123 


13 


.. 65 . 


755 


Where AoDlicatinns Were Processed ^ 

•■ ■ ■ 


Central 
(SFA) 1 


School 
Building 


.School School 
Building* Buildihg' 




Where Quality Assurance Procedures Were 
Processed 


(fehtraV' School 
(SFAJ' Building 


School ! 
Building 


School & 

Gegtral 

»■-».■.- 


' • ■ 

Number of IVPP Applications Prodessed * 

■■ ■ ^' ' i • ^ ■ - J"' ■ - ^ \ ■ 




- A. t 


'. 




• * 


' total Niimber of IVPP Add 11 cat ions 


(NA): 


NA 
; (NA) 


' (13) 


18,258 
(65) . 


78,618 
(45B) • 


IVPP Application Only 
'(Quality Assurance Procedure 1) 


NA 
(NA) 


NA 
(NA) 


1,318 


11,872 ^ 
(43) 


50,9^4 
(316) 


• IVPP Apprication , With. Documentation, 
(Quality Assurance Procedure il) 


: ^ - NA 
(NA) 


' -^NA 
(NA) 


• 499 


6,386 
■ ' (22) 


27,67^ 
(14*2) 


numDer or. uocumeni LonsvSteQcy tnecKs . 
Performed (Quality Assurance 
. Procedure III) 


NA 

• (NA) 


liA* 

(NA) 


• 12 
(2) 


126 
(11) 


(70) 


Number of Document After- Application . 
With Document Consistency Check Performed 
(Quality Assurance P^cedure IV) 


511 


809 
(41) 


- 39 


- 273^ 
(21) 


2,695 . 
,(234) 


, Number of Parent Telephone Conferences 
' Performed (Quality Assurance Procedure V) 


' 378 
(31) 


479 


24 

(6)- 


3B7' 
. (33) 


2,885 • 
(315) . 


Number of Locail Third Party Checks Performed 
(Quality Assurance Prpcedure VI) 


601 
, (35) 


^143 


82 . 
(13) 


312 
(65) 


3,B33 
(l73) 


r Number of State Third Party Checks Performed 
..(Quality Assurance Procedure VII) 


NA 
(NA) 


' NA 
(NA) 


, NA 
. . (NA) . 


NA 

m 


1.933 

{hi) 



^wmmm : 



(1) Numbers shown in parentheses ^present the humber^of schools involved in each Qutility 
- Assurance Procedure ^ 
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: ' APPENDIX C ■ • ' ■ 

MEASUREHENT OF QUALITf ASSURANCE PROCEDURE ^W^^ 

It Is. Important to note that severaV factors complicate the cost 
savings analysis and the calculation of /ji simple indicator of cost 
s'avlng. First, costs and. savings ~occur to. different governmental units. 
In the schoot*m6al programs, all' savings resulting from ap^lcant error . 
reduction accrue to the federal goyerriment. Moreover, cost 'savings for 
the,federal government represent revenue loss for local SFAs. That is, 
the total reimbursement received by the SFA d^icreases because there are 
fewer free and reduced-prlce'meal participants. As a result, when Judged 
strictly from the federal perspective, an^ quality assurance procedure Is 
likely to be cost-effective because requiring the procedure would Involve 
no "costs to, the federal government and Is likely to result In at least 
nominal cost savings. Conversely, from the SFA perspective, no procedure 
could be cost-effective. Any quality control effort 'is likely to Involve 
additional costs. If the procedure discovers or deters mlsreportlngV the 
monetary effect on local SFAs would be revenue loss. Therefore, to rely 
solely on cost savings ratios to evaluate the procedures would be 
administratively naive. However, careful analysis of the costs and 
savings allows ^m'pirtcal explication of these Issues and can form the 
foundation of an Incentive system that would make free and reduced-price ^ 
school meal application quality assurance procedures attractive to both 
%e federal government and local SFAs. 4 ' 

Seaond, there are significant problems In measuring total costs. 
SFAs do not have recordkeeping systems that permit Identification and 
costs associated with ihe 1mplementat16n of quality assurance procedures. 
An extens.lve review of the accounting, systems used by SFAs concluded that 
the only reliable measure of costs would be to have school officials 
maintain records of labor time required to Implement the procedures. This 
Information, In turn, was used to estimate a crude measure of average 
costs of conducting a quality assurance procedure: 
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•where:" . - ' ■ ' -,, ■ 

C^j a Average cost of -1'mple 1n SFA 3 

t ■ -Total time staff members 1n labor category k spend Implementing 
13k procedure -1 In SFA 3 

P ■ a Average total compensation received by staff members In labor 
' Jk category. k In SFA 3 • 

jfi^j a' Number of procedure 1 Implemented 1n SFA 3 

« Total administrative -costs for SFA j 
Pj a Total cost for\adm1n1 strati ve personnel compensation , In SFA J., 

*The cost measure described above develops an average cost estimate 
through the use of measur^ of staff time needed to Implement the 
procedures, number ofvappllcatlons verified, staff compensation rates, and 
the total administrative costs; The cost^measure Incorporates nonlabor 
costs associated with a procedure by multiplying labor costs of a 
procedure by the ratio of total administrative costs for an SFA over total 
administrative personnel compensation. This Is a common bookkeeping 
procedure that avoids the necessity of detailed breakdowns of SFA direct 
and overhead costs. All pilot site SFAs had these data available. 

A limitation Ji^f the cost measure. used Is that It measures only 
average c-osts Jf|d'.cannot address Issues related to marginal costs. 
: Bec'ause of administrative overhead and other sta»;t-up costs, It Is likely 
that the first year of verification is "tfie most expensive and that the 
costs associated with conducting verification in subsequent years will 
progressively decline to a certain minimum. Knowledge of marginal costs 
would potentially permit estimation of the optimal verification rate 
where marginal costs equal marginal benefits'. At such an optimum, total 
savings associated with the procedure would be maximized. Because SFA 
records permitted estimation of o^ily average costs, all 
cost-effectiveness measures will be 1m|il1c1tly Indexed to the 10 percent 
verification rate used In the pilot sites-. 

The average cost measure addresses only per verification costs of 
conducting a procedure; Jt does not address costs associated with 
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Implementing procedures such as staff training, forms development, 
technical assistance, and so on. 'In the Phase H pilot sites, the 
majority of these costs were borne by the contractor and with few 

■ -exceptions did not accrue to SFAs. If adopted Independently, SFAs would 
J. be faced with Implementation and development' costs not captured In the 

h average cost model. However.such Implementation and development costs 
would occur primiarlly In the first year of operation and decline as a 
percentage of total costs In subsequent years. 

By focusing solely on SFA administrative costs, the average cost ' 
model also does not measure costs to applicants: the time and effort 
necessary for applicants to comply with quality assurance procedure " 
requirements Is not Included In the total. Nutritional costs also are 
, not Measured. When a student 1 participation, there are 

potential nutritional costs to the student along with associated cost 
savings to the federal government. The nutritional advantages of the 
school meal program are not necessarily limited to those participants who 
/ are legally eligible. The Phase II design was Incapable of measuring 
these costs. Therefore, measures of cost were necessarily limited to 
average administrative costs accruing to SFAs associated with operating a 
quality assurance procedure. ^ / 

Measurement of costs solely .In terms of average SFA administrative 
expenses could result In a distorted Image of the relatlv^^^^^^^^^^^ 
effectiveness of the procedures. The comparisons of average cost measure 
are biased by making procedures that shift the burden of verification 
from SFAs either to parents (as In the documentation requirement) or to 
V other agencies (as In state-level third-party verifications j appear more 
^ cost-effective than procedures (such as parent conferences) that place 
the burden on SFAs. 

Finally, the average cost model may not adequa^tely represent the 
bookkeeping reality of SFAs. Although the model measures the value/of 
administrative resources absorbed In conducting the procedures, JWMloes 
not necessarily measure total additional administrative costs. In most 
. pilot SFAs, the costs of the procedures were borne by Increasing the 
labor requirements for existing staff and/or cutting back on other staff 
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activities. In only three SFAs, total paid staff time was Increased as a 
result of the procedures. Forms filled put by school staff attempted to 
determine whether other activities had suffered as a result of Imposition 
#f the" procedures. However, a review of forms returned suggested that 
school personnel are unable to articulate the "opportunity costs" of the 
procedures. ' 




APPENDIX 0 . ' » 

• NONRESPONSE Analysis 

This appendix presents Jhe' method used to detect nonresponse bias - 
wUh respect to •receipt>o^^ benefits. The problem addressed Is 

that results of the survey could pbtentlally be biased If households that 
fnlsreport their Income or family size to receive excess benefits have a 
hlgher-^probablllty of refusing to submit to an In-home audit than 
households not receiving excess benefits. The potential for refusal 
bias, however, Is relatively small because the survey experienced only a 
3.25 percent refusal rate.- ' «^ ^ 

Notation . # v; ' • 

R Index variable of response^ status; R « Respondent, R « Refusal* 

B Index variable of excess benefits; B = No excels s benefits, 
B a Excess benefits • 

J Index variable of group membership where groups are constructed 
to maximize the across group varlance^of 8 and minimize the 
variance within group* 

Statistical Procedures 

P(B) Is to be estimated. However, because of potential nonresponse 
bias, E(P(B|R)) / P(B). Therefore, P(B) was estimated Indirectly by the 
equation: 

P(B) =», P(B|R)P(R) + P(BIR)P(R), (1) 

where P(R) and P(R) are the observed response rates for the survey and 
P(B|r) Is to be estimated. J / 

A we1ghted*-1east-squares stepwise discriminate function procedure was. 
employed to divide the survey respondents into 'J' groups, which maximize 
across group variance on excess benefits on the basis of application 
variables. The JJ' groups were constructed by assigning a value to 
each household sampled based on error-prone profile (EPP) score, presence 



or absence of reported wage Income, and presence or absence. of reported 
welfare 'Income. ' , • f- 

, , ■ The sample was divided Into lO'groups (0) based on Intervals of Q. f>le ' 
make the assumpU on that: \ \. v , ' .. 

" , : V^^^^^ r (2) 

- ' ■ . ■ ' V ■ and ' . ' . '■' ■ 

; p(R|B,j) n p(r|b). . ' (3) 

That Is, the relation of R and B Is constant across categories of. 0. 



. Because B and B are mutually exclusive and- exhaustive categories: 

V ' >(B|3) = 1 - P(BlJ) (4) 
>and therefore, \ V 

\ , 1- m\\) \ ; 

. . p(B|j) * : ' (5) 

Multiplying both sides by P(R|Bj/P(RjB) results In: 

P(R|B\/P(RiB^ n PfRlBl- PfRlf) X Pdl'j) ~ (6) 

, ' P(R|B) x^P{&lj) " 

from Assumption 3: 

P(R|B) X P(B|j) = P(RB|j) and 

P(R|B) X P(B|j) » P(RB|j). ' ■ (7) 

■ ■ i . . 

Substituting these values Into Equation 6 obtains: j 

Solving for P(RBl3): 

' P{RBU) - P{R|B) - ^{1^ P(RF|j) . (9) 



Thus, there -Is a linear relationship between P(Rl,| D^and ^P(RB | 3) that 
jls a function of P(R|B) and P(R|i). ThU relat1q(ish1p can he expressed • 
1n standard notation: ■ 

- ■ ■ .■ , ■ ^ 

R(RBji) . a + /9(RB|3), ' . (10) , 

Where: 

an P{R|B) 

^p(RiBi . 

: P(r|T) • . " 

a and j9 was estimated using OLS methods and assumptions: - 

> a + bxj + e, ' (11) 

where: ' ' - 

. . yi 1s P(Rb|J) and 
Xj Is P(RB|j). 

The abta^ed values of a and b were used to obtain estimates of. P(R1b) 
and P(R|B) from Equation 9: ^ 

. P(R|B) - a ■ (12) 

V / 'p(rIb) ^ 
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. The analysis revealed no strong, consistent, or statistically significant 
refusal bias. In fact, households with low EPP scores (Indicating a low 
probability of receipt of excess benefits) were slightly more likely to refuse 
an Interview than households with medium or high EPP scores. We conclude from 
this analysis there Is no evidenc;? of response bias In the In-home audit data. 



' ■ .." " ^ END .NOTES: . , : . . .. 

" ■ . ■ . /■ . ■■•^ .■ ■ ' 

y Refusals are defined as households selected for In-home; fiudlts who 
refused to participate In an Interview. There were 83 'such refusals. 
Households that could not be located because they had. moved from the * 
areas where the In-home- audits were conducted were not cons-ldered 
• refusals. . ■ .. . 




